Istio Implementation

1. Must-read

Architecture of Istiod

2. Getting Started

2.1. Components

Need to understand the Istio components

Envoy: sidecar proxies for microservices to handle ingress/egress traffic between services in the cluster and from a service to external services.
Istiod: Istio control plane, which provides service discovery, configuration and certificate management <- This is a modular monolith
1. Pilot: Responsible for configuring the proxies at runtime
2. Citadel: Responsible for certificate issuance and rotation.
3. Gallery: Responsible for validating, ingesting, aggregating, transforming, and distributing config within Istio
Operator: The component provides user friendly options to operate the Istio service mesh. The role is management of istio components (istiod, istio-ingressgateway, and istio-egressgateway) with IstioOperator Custom Resource, which is also same role of istioctl (Istio Operator Inatall)

Memo

Istiod from microservices to a single binary istiod in 2020: <- Very interesting story!

Istio’s control plane is, itself, a modern, cloud-native application. Thus, it was built from the start as a set of microservices. Individual Istio components like service discovery (Pilot), configuration (Galley), certificate generation (Citadel) and extensibility (Mixer) were all written and deployed as separate microservices.

Having established that many of the common benefits of microservices didn’t apply to the Istio control plane, we decided to unify them into a single binary: istiod (the ’d’ is for daemon). 1. Pilot seems to represent istiod: https://github.com/istio/istio/blob/master/architecture/networking/pilot.md <- You must read this to know the details about Istiod 1. Ingress: In addition to VirtualService and Gateway, Istio supports the Ingress core resource type. Like CRDs, the Ingress controller implements ConfigStore, but a bit differently. Ingress resources are converted on the fly to VirtualService and Gateway, read more 1. Gateway API: Gateway (referring to the Kubernetes API, not the same-named Istio type) works very similarly to Ingress. The Gateway controller also coverts Gateway API types into VirtualService and Gateway, implementing the ConfigStore interface.

2.2. Repository

Better to know the structure of the repository! Please read https://github.com/istio/istio/tree/1.19.0#repositories

3. Installation

3.1. Install with Istio Operator

Install Istio Operator

istioctl operator init

istio-operator Deployment will be deployed in istio-operator Namespace.

Install Istio with IstioOperator CR

kubectl apply -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  profile: demo
EOF

3.2. Install with `istioctl`

istioctl profile list
istioctl profile dump demo # demo is one of the profiles

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    base:
      enabled: true
    cni:
      enabled: false
    egressGateways:
    - enabled: true
      k8s:
        resources:
          requests:
            cpu: 10m
            memory: 40Mi
      name: istio-egressgateway
    ingressGateways:
    - enabled: true
      k8s:
        resources:
          requests:
            cpu: 10m
            memory: 40Mi
        service:
          ports:
          - name: status-port
            port: 15021
            targetPort: 15021
          - name: http2
            port: 80
            targetPort: 8080
          - name: https
            port: 443
            targetPort: 8443
          - name: tcp
            port: 31400
            targetPort: 31400
          - name: tls
            port: 15443
            targetPort: 15443
      name: istio-ingressgateway
    istiodRemote:
      enabled: false
    pilot:
      enabled: true
      k8s:
        env:
        - name: PILOT_TRACE_SAMPLING
          value: "100"
        resources:
          requests:
            cpu: 10m
            memory: 100Mi
  hub: docker.io/istio
  meshConfig:
    accessLogFile: /dev/stdout
    defaultConfig:
      proxyMetadata: {}
    enablePrometheusMerge: true
    extensionProviders:
    - envoyOtelAls:
        port: 4317
        service: opentelemetry-collector.istio-system.svc.cluster.local
      name: otel
    - name: skywalking
      skywalking:
        port: 11800
        service: tracing.istio-system.svc.cluster.local
    - name: otel-tracing
      opentelemetry:
        port: 4317
        service: opentelemetry-collector.otel-collector.svc.cluster.local
  profile: demo
  tag: 1.19.0
  values:
    base:
      enableCRDTemplates: false
      validationURL: ""
    defaultRevision: ""
    gateways:
      istio-egressgateway:
        autoscaleEnabled: false
        env: {}
        name: istio-egressgateway
        secretVolumes:
        - mountPath: /etc/istio/egressgateway-certs
          name: egressgateway-certs
          secretName: istio-egressgateway-certs
        - mountPath: /etc/istio/egressgateway-ca-certs
          name: egressgateway-ca-certs
          secretName: istio-egressgateway-ca-certs
        type: ClusterIP
      istio-ingressgateway:
        autoscaleEnabled: false
        env: {}
        name: istio-ingressgateway
        secretVolumes:
        - mountPath: /etc/istio/ingressgateway-certs
          name: ingressgateway-certs
          secretName: istio-ingressgateway-certs
        - mountPath: /etc/istio/ingressgateway-ca-certs
          name: ingressgateway-ca-certs
          secretName: istio-ingressgateway-ca-certs
        type: LoadBalancer
    global:
      configValidation: true
      defaultNodeSelector: {}
      defaultPodDisruptionBudget:
        enabled: true
      defaultResources:
        requests:
          cpu: 10m
      imagePullPolicy: ""
      imagePullSecrets: []
      istioNamespace: istio-system
      istiod:
        enableAnalysis: false
      jwtPolicy: third-party-jwt
      logAsJson: false
      logging:
        level: default:info
      meshNetworks: {}
      mountMtlsCerts: false
      multiCluster:
        clusterName: ""
        enabled: false
      network: ""
      omitSidecarInjectorConfigMap: false
      oneNamespace: false
      operatorManageWebhooks: false
      pilotCertProvider: istiod
      priorityClassName: ""
      proxy:
        autoInject: enabled
        clusterDomain: cluster.local
        componentLogLevel: misc:error
        enableCoreDump: false
        excludeIPRanges: ""
        excludeInboundPorts: ""
        excludeOutboundPorts: ""
        image: proxyv2
        includeIPRanges: '*'
        logLevel: warning
        privileged: false
        readinessFailureThreshold: 30
        readinessInitialDelaySeconds: 1
        readinessPeriodSeconds: 2
        resources:
          limits:
            cpu: 2000m
            memory: 1024Mi
          requests:
            cpu: 10m
            memory: 40Mi
        statusPort: 15020
        tracer: zipkin
      proxy_init:
        image: proxyv2
      sds:
        token:
          aud: istio-ca
      sts:
        servicePort: 0
      tracer:
        datadog: {}
        lightstep: {}
        stackdriver: {}
        zipkin: {}
      useMCP: false
    istiodRemote:
      injectionURL: ""
    pilot:
      autoscaleEnabled: false
      autoscaleMax: 5
      autoscaleMin: 1
      configMap: true
      cpu:
        targetAverageUtilization: 80
      env: {}
      image: pilot
      keepaliveMaxServerConnectionAge: 30m
      nodeSelector: {}
      podLabels: {}
      replicaCount: 1
      traceSampling: 1
    telemetry:
      enabled: true
      v2:
        enabled: true
        metadataExchange:
          wasmEnabled: false
        prometheus:
          enabled: true
          wasmEnabled: false
        stackdriver:
          configOverride: {}
          enabled: false
          logging: false
          monitoring: false
          topology: false

What istioctl install (you can install with -p demo to specify a profile) does is to create an IstioOperator Custom Resource and Istio Controller does the actual work.

Relationship between the CLI and controller:

The CLI and controller share the same API and codebase for generating manifests from the API. You can think of the controller as the CLI command istioctl install running in a loop in a pod in the cluster and using the config from the in-cluster IstioOperator custom resource (CR).

4. Code Reading

4.1. pilot-discovery (istiod)

pilot/cmd/pilot-discovery/main.go
1. pilot/cmd/pilot-discovery/app/cmd.go
```
discoveryServer, err := bootstrap.NewServer(serverArgs)
```
bootstrap.NewServer
1. aggregate.NewController
2. Initialize Server -> s
3. xds.NewDiscoveryServer
4. initServers
5. initKubeClient
6. initMeshConfiguration
7. initMeshNetworks
8. initMeshHandlers
9. s.environment.InitNetworksManager
10. maybeCreateCA
11. initControllers
12. s.XDSServer.InitGenerators
13. s.initWorkloadTrustBundle
14. s.initIstiodCerts
15. s.initSecureDiscoveryService
16. s.initSecureWebhookServer
17. s.initSidecarInjector
18. s.initConfigValidation
19. s.initRegistryEventHandlers
20. s.initDiscoveryService

4.2. Ingress

pilot/pkg/config/kube/ingress/controller.go

Ingress、Gateway、VirtualServiceを管理してるけど、pilotとは？ -> Istiod内のComponentでTraffic Managementなどを担当する主要コンポーネント

// In 1.0, the Gateway is defined in the namespace where the actual controller runs, and needs to be managed by
// user.
// The gateway is named by appending "-istio-autogenerated-k8s-ingress" to the name of the ingress.
//
// Currently the gateway namespace is hardcoded to istio-system (model.IstioIngressNamespace)
//
// VirtualServices are also auto-generated in the model.IstioIngressNamespace.
//
// The sync of Ingress objects to IP is done by status.go
// the 'ingress service' name is used to get the IP of the Service
// If ingress service is empty, it falls back to NodeExternalIP list, selected using the labels.
// This is using 'namespace' of pilot - but seems to be broken (never worked), since it uses Pilot's pod labels
// instead of the ingress labels.

// Follows mesh.IngressControllerMode setting to enable - OFF|STRICT|DEFAULT.
// STRICT requires "kubernetes.io/ingress.class" == mesh.IngressClass
// DEFAULT allows Ingress without explicit class.

// In 1.1:
// - K8S_INGRESS_NS - namespace of the Gateway that will act as ingress.
// - labels of the gateway set to "app=ingressgateway" for node_port, service set to 'ingressgateway' (matching default install)
//   If we need more flexibility - we can add it (but likely we'll deprecate ingress support first)
// -

1. pilot/pkg/config/kube/gateway/controller.g: This is for Gateway API

Local Development

https://github.com/istio/istio/wiki/Preparing-for-Development

https://github.com/istio/istio/wiki/Using-the-Code-Base

Clone

mkdir -p $GOPATH/src/istio.io/istio
git clone https://github.com/istio/istio $GOPATH/src/istio.io/istio
cd $GOPATH/src/istio.io/istio

Set env var

USER=nakamasato # in my case

# This defines the docker hub to use when running integration tests and building docker images
# eg: HUB="docker.io/istio", HUB="gcr.io/istio-testing"
export HUB="docker.io/$USER"

# This defines the docker tag to use when running integration tests and
# building docker images to be your user id. You may also set this variable
# this to any other legitimate docker tag.
export TAG=$USER

# This defines a shortcut to change directories to $HOME/istio.io
export ISTIO=$GOPATH/src/istio.io/istio

Memo

Overview

istio-ingressgateway, istio-egressgateway
1. Helmはここ
2. imageはproxyv2が使われてる
3. proxyv2 はpilot-agentにも使われているからどういう違いかが不明
pilot-agent sidecar
1. proxyv2
2. Dockerfile.proxyv2: Entrypoint: "/usr/local/bin/pilot-agent"
3. InjectされるSidecarのことが pilot-agentと呼ばれているっぽい
4. The code for sidecar injection: https://github.com/istio/istio/blob/1.19.0/pkg/kube/inject/inject.go
5. main.go: https://github.com/istio/istio/blob/1.19.0/pilot/cmd/pilot-agent/main.go
pilot-discovery controller <- istiod Deployment
1. Docker: https://github.com/istio/istio/blob/1.19.0/pilot/docker/Dockerfile.pilot
2. main.go: Entrypoint of istiod server
  1. pilot/cmd/pilot-discovery/main.go
  2. pilot/cmd/pilot-discovery/app/cmd.go
```
discoveryServer, err := bootstrap.NewServer(serverArgs)
```
3. From Logs, https://github.com/istio/istio/blob/1.19.0/pilot/pkg/bootstrap/server.go is the istiod server initialization.
  1. bootstrap.NewServer
    1. initializing istiod admin server: initServers
    2. initIstiodAdminServer
    3. initControllers
    4. initializing secure discovery service: initSecureDiscoveryService
    5. initializing secure webhook server for istiod webhooks: initSecureWebhookServer
    6. initializing sidecar injector: initSidecarInjector
    7. initializing config validator: initConfigValidation
    8. initializing registry event handlers: initRegistryEventHandlers
    9. starting discovery service: initDiscoveryService

FAQ

How sidecar container is injected -> https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/#automatic-sidecar-injection > Sidecars can be automatically added to applicable Kubernetes pods using a mutating webhook admission controller provided by Istio.

Implementation: https://github.com/istio/istio/blob/1.19.0/pkg/kube/inject/webhook.go

Istio Implementation

1. Must-read

2. Getting Started

2.1. Components

2.2. Repository

3. Installation

3.1. Install with Istio Operator

3.2. Install with istioctl

4. Code Reading

4.1. pilot-discovery (istiod)

4.2. Ingress

Local Development

Memo

Overview

FAQ

Ref

3.2. Install with `istioctl`