HTTP_X Forwarded For NGINX Ingress not working

Question

I have a simple application running inside AKS. I'm trying to get the IP of the source user when accessing the application. I have created an ingress for the application. When I curl -I https://example-app.xyz I see the source IP in the logs of NGINX ingress controller (It just shows as client IP in the logs). My requirement is to do something like below.

annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
               if ($http_x_forwarded_for != "<my-ip-address>") {
               return 301 https://example-app2.xyz;
                    }

I have added the below to the configmap of NGINX as well.

data:
  use-forwarded-headers: "true"

But all requests are forwarded to https://example-app2.xyz now. I can see the source IP in the logs in NGINX ingress controller. When I curl -I https://example-app.xyz it doesn't even show X-Forwarded- header.