x-forwarded-for header is not shown on Envoy response

Question

I have a working LDS.yaml file, and I want to add x-forwarded-for in the header since in the envoy documentation says it should be shown if Envoy works as edge server. But when I use this best practice configuration https://www.envoyproxy.io/docs/envoy/latest/configuration/best_practices/edge, the header x-forwarded-for is not shown

Here is my LDS.yaml

version_info: "0"
resources:
  # not enforcing https
  - name: "http_listener"
    "@type": type.googleapis.com/envoy.api.v2.Listener  
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 80
    filter_chains:
      filters:
        - name: envoy.filters.network.http_connection_manager
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            server_name: foo.example.com
            request_timeout: 60s
            use_remote_address: true # enabling x-forward-for
            xff_num_trusted_hops: 1 # only show 1 x-forward-for
            always_set_request_id_in_response: true # enable x-request-id
            codec_type: auto
            stat_prefix: ingress_http
            http_filters:
              - name: envoy.filters.http.router
                typed_config: {}
            route_config:
...

Is there anything wrong?

Thank you!

Breedly · Answer 1 · 2022-01-14T20:45:14.543

use_remote actually disables the usage of X-FORWARDED-FOR

If set to true, the connection manager will use the real remote address of the client connection when determining internal versus external origin and manipulating various headers. If set to false or absent, the connection manager will use the x-forwarded-for HTTP header. See the documentation for x-forwarded-for, x-envoy-internal, and x-envoy-external-address for more information.

xff_num_trusted_hops should be used when use_remote is false.

Google Documents fairly well what the X-Forwarded-For header looks and acts like on their platform (Source).

X-Forwarded-For: <client-ip>,<load-balancer-ip>

I'm not sure if this is one or two jumps, but if the request comes from the Google LB you can "trust" (?) that client-ip is the actual IP that communicated with the Google LB.

I myself am struggling to have envoy log the X-FORWARDED-FOR header in its entirety.

When I use use_remote: true I get the IP address of the load balancer. When I use use_remote: false I get -. In either case I know these are not the X-FORWARDED-FOR headers I'm looking for. (Envoy: Access logging)

I wanted to see if I were actually receiving X-Forwarded-For headers so I edited my access log block to log a few different instances of X-Forwarded-For and my own custom header:

#add this to "envoy.filters.network.http_connection_manager" config
access_log:
  - name: envoy.access_loggers.stdout
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
      log_format:
        json_format: 
          a: "%REQ(X-Forwarded-For)%"
          b: "%REQ(X-FORWARDED-FOR)%"
          mine: "%REQ(X-MY-HEADER)%"

curl -k http://my-envoy.com:8080/ready -v -H 'X-MY-HEADER: mine'

{"a":null,"b":null,"mine":"mine"}

So in my case the proxy in front of envoy is not setting the X-Forwarded-For header by default.

I was able to verify this by then making this curl request:

curl -k http://my-envoy.com:8080/ready -v -H 'X-MY-HEADER: mine' -H 'X-Forwarded-For: 127.0.0.1'

{"b":"127.0.0.1","a":"127.0.0.1","mine":"mine"}

So the access log is case insensitive and it will log the header X-Forwarded-For header if it does exist.

x-forwarded-for header is not shown on Envoy response

1 Answers1