10

Our GKE cluster is shared to multiple teams in company. Each team can have different public domain (and hence want to have different CA cert setup and also different ingress gateway controller). How to do that in Istio? All the tutorial/introduction articles in Istio's website are using a shared ingress gateway. See the example shared ingress gateway that comes installed by istio-1.0.0: https://istio.io/docs/tasks/traffic-management/secure-ingress/ 

spec:
  selector:
    istio: ingressgateway # use istio default ingress gateway
Agung Pratama
  • 3,666
  • 7
  • 36
  • 77

5 Answers5

18

Okay, I found the answer after looking at the code of Istio installation via helm. So, basically the istio have an official way (but not really documented in their readme.md file) to add additional gateway (ingress and egress gateway). I know that because I found this yaml file in their github repo and read the comment (also looking at the gateway chart template code for the spec and its logic).

So, I solved this by, for example, defining this values-custom-gateway.yaml file:

# Gateways Configuration
# By default (if enabled) a pair of Ingress and Egress Gateways will be created for the mesh.
# You can add more gateways in addition to the defaults but make sure those are uniquely named
# and that NodePorts are not conflicting.
# Disable specifc gateway by setting the `enabled` to false.
#
gateways:
  enabled: true

  agung-ingressgateway:
    namespace: agung-ns
    enabled: true
    labels:
      app: agung-istio-ingressgateway
      istio: agung-ingressgateway
    replicaCount: 1
    autoscaleMin: 1
    autoscaleMax: 2
    resources: {}
      # limits:
      #  cpu: 100m
      #  memory: 128Mi
      #requests:
      #  cpu: 1800m
      #  memory: 256Mi

    loadBalancerIP: ""
    serviceAnnotations: {}
    type: LoadBalancer #change to NodePort, ClusterIP or LoadBalancer if need be

    ports:
      ## You can add custom gateway ports
    - port: 80
      targetPort: 80
      name: http2
      # nodePort: 31380
    - port: 443
      name: https
      # nodePort: 31390
    - port: 31400
      name: tcp
    secretVolumes:
    - name: ingressgateway-certs
      secretName: istio-ingressgateway-certs
      mountPath: /etc/istio/ingressgateway-certs
    - name: ingressgateway-ca-certs
      secretName: istio-ingressgateway-ca-certs
      mountPath: /etc/istio/ingressgateway-ca-certs

If you take a look at yaml file above, I specified the namespace other than istio-system ns. In this case, we can have a way to customize the TLS and ca cert being used by our custom gateway. Also the agung-ingressgateway as the holder of the custom gateway controller spec is used as the gateway controller's name.

Then, i just install the istio via helm upgrade --install so that helm can intelligently upgrade the istio with additional gateway.

helm upgrade my-istio-release-name <istio-chart-folder> --install

Once it upgrades successfully, I can specify custom selector to my Gateway:

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: agung-gateway
  namespace: agung-ns
spec:
  selector:
    app: agung-istio-ingressgateway # use custom gateway
    # istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - "*"
bossylobster
  • 9,993
  • 1
  • 42
  • 61
Agung Pratama
  • 3,666
  • 7
  • 36
  • 77
  • @4c74356b41 that would be from the secret `ingressgateway-ca-certs` mounted to that volume. See my snippet that provision the gateway. there is `secretVolumes` definition there – Agung Pratama Feb 14 '19 at 10:48
  • 1
    hey, where would it get ca certificates from? also, i dont really understand, if I upgrade existing istio release, it deletes pilot\mixer\etc, if I try to create a new release from custom-gateway it just blows up. yeah, i saw that, but what would create that secret? its not in any namespace (including istio-system) – 4c74356b41 Feb 14 '19 at 10:51
  • yes, i see that istio default installation doesn't have a mechanism to do rolling upgrade. Meaning that when you change the istio image version, it will terminate existing control plane and spawn a new one. I myself currently in preparation to customize the helm installation for doing upgrade preparation, without downtime (strill trying to) – Agung Pratama Feb 14 '19 at 10:55
  • ` if I try to create a new release from custom-gateway` you don't create a new release, just do `helm upgrade --install`. – Agung Pratama Feb 14 '19 at 10:56
  • 1
    like I said, it will delete the pods in the istio-system namespace – 4c74356b41 Feb 14 '19 at 11:05
  • @ 4c74356b41 no it won't kill them as long as you don't change any configuration that affect the pods in istio-system, since you only change the gateway configuration. I've been using this way since 6 months ago – Agung Pratama Feb 14 '19 at 12:20
  • 1
    @4c74356b41 perhaps for better understanding on your problem, create an SO question for this is good idea :) – Agung Pratama Feb 14 '19 at 12:36
  • @AgungPratama you said "I specified the namespace other than istio-system ns". How can i create ingress gateway in different namespaces. I mean actually i need to create ingress gateway in cicdapi-testing namespace and cicdapi-production namespace, how can i achieve this through this values.yaml ? – Gabriel Wu Jul 02 '19 at 01:14
4

This is what I use in Istio 1.4.

To generate the new istio-ingressgateway Deployment, Service and ServiceAccount in your own namespace (bookinfo in this example)

helm template install/kubernetes/helm/istio/ \
  --namespace bookinfo \
  --set global.istioNamespace=istio-system \
  -x charts/gateways/templates/deployment.yaml \
  -x charts/gateways/templates/service.yaml  \
  -x charts/gateways/templates/serviceaccount.yaml \
  --set gateways.istio-ingressgateway.enabled=true \
  --set gateways.istio-egressgateway.enabled=false \
  --set gateways.istio-ingressgateway.labels.app=custom-istio-ingressgateway \
  --set gateways.istio-ingressgateway.labels.istio=custom-ingressgateway \
  > customingress.yaml

then, apply the generated file:

kubectl apply -f customingress.yaml

Now, you can refer to this from your Gateway resource like:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway
spec:
  selector:
    istio: custom-ingressgateway # use the CUSTOM istio controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

You can set custom Service annotations by adding it to the helm template command like this:

 --set gateways.istio-ingressgateway.serviceAnnotations.'service\.kubernetes\.io/ibm-load-balancer-cloud-provider-ip-type'=private \
Ram Vennam
  • 3,536
  • 1
  • 12
  • 19
3

I tried this and worked:

---
# Source: istio/charts/gateways/templates/serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: beta-ingressgateway-service-account
  namespace: beta
  labels:
    app: ingressgateway-beta
---

---
# Source: istio/charts/gateways/templates/clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  labels:
    app: gateways
  name: ingressgateway-beta
rules:
- apiGroups: ["extensions"]
  resources: ["thirdpartyresources", "virtualservices", "destinationrules", "gateways"]
  verbs: ["get", "watch", "list", "update"]
---

---
# Source: istio/charts/gateways/templates/clusterrolebindings.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ingressgateway-beta
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingressgateway-beta
subjects:
  - kind: ServiceAccount
    name: beta-ingressgateway-service-account
    namespace: beta
---

---
# Source: istio/charts/gateways/templates/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: ingressgateway-beta
  namespace: beta
  annotations:
  labels:
    istio: ingressgateway-beta
spec:
  type: LoadBalancer
  selector:
    istio: ingressgateway-beta
  ports:
    -
      name: http
      port: 80
      targetPort: 80
    -
      name: https
      port: 443
      targetPort: 443
---

---
# Source: istio/charts/gateways/templates/deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ingressgateway-beta
  namespace: beta
  labels:
    istio: ingressgateway-beta
spec:
  replicas: 1
  template:
    metadata:
      labels:
        istio: ingressgateway-beta
      annotations:
        sidecar.istio.io/inject: "false"
        scheduler.alpha.kubernetes.io/critical-pod: ""
    spec:
      serviceAccountName: beta-ingressgateway-service-account
      tolerations:
      - key: "env"
        operator: "Equal"
        value: "beta"
        effect: "NoSchedule"
      nodeSelector:
        env: beta
      containers:
        - name: istio-proxy
          image: "ISTIO_PROXY_IMAGE"
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
            - containerPort: 443
          args:
          - proxy
          - router
          - -v
          - "2"
          - --discoveryRefreshDelay
          - '1s' #discoveryRefreshDelay
          - --drainDuration
          - '45s' #drainDuration
          - --parentShutdownDuration
          - '1m0s' #parentShutdownDuration
          - --connectTimeout
          - '10s' #connectTimeout
          - --serviceCluster
          - ingressgateway-beta
          - --zipkinAddress
          - zipkin.istio-system:9411
          - --proxyAdminPort
          - "15000"
          - --controlPlaneAuthPolicy
          - NONE
          - --discoveryAddress
          - istio-pilot.istio-system:8080
          resources:
            requests:
              cpu: 10m
          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          - name: INSTANCE_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.podIP
          - name: ISTIO_META_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          volumeMounts:
          - name: istio-certs
            mountPath: /etc/certs
            readOnly: true
          - name: ingressgateway-beta-certs
            mountPath: "/etc/istio/ingressgateway-beta-certs"
            readOnly: true
          - name: ingressgateway-beta-ca-certs
            mountPath: "/etc/istio/ingressgateway-beta-ca-certs"
            readOnly: true
      volumes:
      - name: istio-certs
        secret:
          secretName: istio.beta-ingressgateway-service-account
          optional: true
      - name: ingressgateway-beta-certs
        secret:
          secretName: "istio-ingressgateway-beta-certs"
          optional: true
      - name: ingressgateway-beta-ca-certs
        secret:
          secretName: "istio-ingressgateway-beta-ca-certs"
          optional: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x
---

---
# Source: istio/charts/gateways/templates/autoscale.yaml
# Source: istio/charts/gateways/templates/autoscale.yaml

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
    name: ingressgateway-beta
    namespace: beta
spec:
    maxReplicas: 5
    minReplicas: 1
    scaleTargetRef:
      apiVersion: apps/v1beta1
      kind: Deployment
      name: ingressgateway-beta
    metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 80
---

remember to replace ISTIO_PROXY_IMAGE, nodeSelector and tolerations

xring
  • 727
  • 2
  • 8
  • 29
0

In fact, it is very simple. Istio's ingress is just a regular Kubernetes Service of "Load Balancer" type. So, if you want to create additional IngresGateway, you can just apply your service (you can put any ports you want):

apiVersion: v1
kind: Service
metadata:
  name: istio-ingressgateway-custom
  namespace: istio-system
  annotations:
  labels:
    chart: gateways-1.0.5
    release: istio
    heritage: Tiller
    app: istio-ingressgateway
    istio: ingressgateway
spec:
  type: LoadBalancer
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  ports:
    -
      name: http2
      nodePort: 31381
      port: 80
      targetPort: 80
    -
      name: https
      nodePort: 31391
      port: 443
      targetPort: 555
    -
      name: tcp
      nodePort: 31401
      port: 31400
    -
      name: tcp-pilot-grpc-tls
      port: 15011
      targetPort: 15011
    -
      name: tcp-citadel-grpc-tls
      port: 8060
      targetPort: 8060
    -
      name: tcp-dns-tls
      port: 853
      targetPort: 853
    -
      name: http2-prometheus
      port: 15030
      targetPort: 15030
    -
      name: http2-grafana
      port: 15031
      targetPort: 15031
---

Considering you have this in file named "customingress.yaml", then you apply this using command:

kubectl apply -f customingress.yaml
Illidan
  • 4,047
  • 3
  • 39
  • 49
  • 3
    No, istio ingress gateway is not a kube service/LB, it is basically a deployment that has istio service running (an istio container, with no side car), can be exposed to public by kube service/LB. It is similar to nginx ingress controller – Agung Pratama Jan 11 '19 at 13:11
  • Istio "ingressgateway", at least in version 1.0.5. is a regular K8s Service. I personally tried the yaml above and it does exactly what is expected: it created for me additional ELB (exposed on dedicated IP) and I was able to route traffic from both ELBs to the same Gateway. – Illidan Jan 11 '19 at 17:15
  • 1
    Yes, that’s because you’re using a shared gateway deployment in istio-system with label `istio: ingressgateway`. The question asking how to create a dedicated one, the one that you can customize the TLS secret etc. Actually i have found the answer by myself here https://stackoverflow.com/a/51840872/476917 – Agung Pratama Jan 12 '19 at 04:48
-2

My guess, I have not try it yet:

  1. Create multiple istio-ingressgateway deployment with differnet labels, like: istio: ingressgateway1, istio: ingressgateway2, ..., and different tls keys.

  2. Create multiple Gateways to use the different istio-ingressgateways.

  3. Create multiple Virtualservice to use the different Gateways.

Kun Li
  • 2,570
  • 10
  • 15