AWS - x-real-ip is ip of nginx-ingress-controller

Question

I currently have the following problem. I have a backend that is behind an nginx-ingress controller used as load balancer in aws. Usually i should get the users real ip by either the header x-forwarded-for or x-real-ip. However this ip always points to the ip of the ingress controller in my case. I use terraform to setup my architecture.

This is my current configuration

resource "helm_release" "ingress_nginx" {
  name             = "ingress-nginx"
  namespace        = "ingress"
  create_namespace = true
  chart            = "ingress-nginx"
  version          = "4.0.13"
  repository       = "https://kubernetes.github.io/ingress-nginx"

  values = [
    <<-EOF
    controller:
      service:
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-type: nlb
      admissionWebhooks:
        enabled: false
    EOF
  ]
}

data "kubernetes_service" "ingress_nginx" {
  metadata {
    name      = "ingress-nginx-controller"
    namespace = helm_release.ingress_nginx.metadata[0].namespace
  }
}

data "aws_lb" "ingress_nginx" {
  name = regex(
    "(^[^-]+)",
    data.kubernetes_service.ingress_nginx.status[0].load_balancer[0].ingress[0].hostname
  )[0]
}

output "lb" {
  value = data.aws_lb.ingress_nginx.name
}`

Does anybody know how to fix the header in this config?

AFAIK, since it seems you are using an NLB, you cannot get the headers from the Layer 7. — Marko E, Feb 18 '22 at 07:09

score 0 · Answer 1 · edited May 17 '22 at 01:18

To get the real IP address in the header both the ingress controller and the NLB (Network Load Balancer) need to use Proxy Protocol. To do this, in terraform you need to configure the ingress controller with the proxy-protocol option:

resource "helm_release" "ingress_nginx" {
  name             = "ingress-nginx"
  namespace        = "ingress"
  create_namespace = true
  chart            = "ingress-nginx"
  version          = "4.0.13"
  repository       = "https://kubernetes.github.io/ingress-nginx"

  values = [
    <<-EOF
    controller:
      service:
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-type: nlb
      config:
        use-forwarded-headers: true
        use-proxy-protocol: true
        enable-real-ip: true
      admissionWebhooks:
        enabled: false
    EOF
  ]
}

data "kubernetes_service" "ingress_nginx" {
  metadata {
    name      = "ingress-nginx-controller"
    namespace = helm_release.ingress_nginx.metadata[0].namespace
  }
}

data "aws_lb" "ingress_nginx" {
  name = regex(
    "(^[^-]+)",
    data.kubernetes_service.ingress_nginx.status[0].load_balancer[0].ingress[0].hostname
  )[0]
}

output "lb" {
  value = data.aws_lb.ingress_nginx.name
}

Now, it seems that currently there is no way to get the NLB to use proxy protocol by annotating the service.

There are some issues in github mentioning that the following annotations would solve it:

*service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip"*

and

*service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"*

However, these have not worked for me. So, I have enabled proxy protocol manually though the AWS Console by doing the following:

AWS console --> EC2 --> Target groups under load balancing --> Select Target groups associated with your NLB --> Attributes --> Enable proxy protocol 2

I guess that in terraform the optimal solution would be to explicitly create a Network Load Balancer with the correct target group and associate it with the ingress controller service.

Doing this, now I get the correct IP addresses of the clients connecting to the ingress controller.

To use an annotation (and avoid this manual step for activating Proxy Protocol on the AWS Network Load Balancer), you need to have the AWS Load Balancer Controller setup in your cluster : https://github.com/kubernetes/kubernetes/issues/101608#issuecomment-847438014 — lboix, Jun 09 '22 at 21:13

AWS - x-real-ip is ip of nginx-ingress-controller

1 Answers1