Preserve client IP in GKE

Question

I have a backend application behind an nginx ingress controller in a GKE cluster and I want to whitelist a certain IP only to access it. I added to the associated ingress this annotation:

nginx.ingress.kubernetes.io/whitelist-source-range: "my-ip/32"

I also have the externalTrafficPolicy set to Local in the ingress controller service.

The issue is that when I hit my application it always return 403 Forbidden And in the ingress controller logs when I hit the application it logs access forbidden by rule, client: 127.0.0.1, server: my-appliaction.domain.ext which means that the client IP is not forwarded to the ingress controller. I am using GCP GKE.

Did I miss something?

Thanks in advance.

score 0 · Answer 1 · answered Mar 07 '23 at 15:19

I used the following nginx ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.5.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  allow-snippet-annotations: "true"
  enable-real-ip: "true"
  use-forwarded-headers: "true"
  proxy-real-ip-cidr: "<pods_cidr>,<services_cidr>,<load_balance_ip>/32"
  use-proxy-protocol: "false"

And added the statement externalTrafficPolicy: Local on nginx Service that assign the load balance:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.5.1
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  externalTrafficPolicy: Local
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  ports:
    - appProtocol: https
      name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer
  loadBalancerIP: <load_balance_ip>

Then, I also configured ip-masq-agent with the follow ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ip-masq-agent
  namespace: kube-system
data:
  config: |
    nonMasqueradeCIDRs:
      - <load_balance_ip>/32
      - <pods_cidr>
      - <services_cidr>
    masqLinkLocal: false
    resyncInterval: 30s

So, I deleted the DaemonSet ip-masq-agent and gke automatic recreated it.

After that, I got my gke cluster working as expected. And used with success nginx.ingress.kubernetes.io/whitelist-source-range on Ingress.

You can find more information about ip-masq-agent on gke accessing https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent

Preserve client IP in GKE

1 Answers1