1

I have a situation with ip_hash on nginx for sticky session where it always direct the traffic to one server only. The nginx is running behind a Google Cloud load balancer, from nginx it hits my app's GC Kubernetes load balancer before actually going to one of the pod.

I have a thought that all users' IP was masked by GC load balancer so that nginx recognises them as one source.

I was told that we can add user's original IP to nginx's header so that the load balancer able to recognise them as different users. How can I approach one? Or is there another solution?

spondbob
  • 123
  • 1
  • 5

3 Answers3

1

If you are using Network load balancing (with target pools) the load balancer keeps the IP. What happens is that Kubernetes is changing the source IP with the cluster/node IPs.

Kubernetes has a feature to preserve the client source IP. You can check in the docs for how to preserve the client source IP in the services with Type=LoadBalancer (Network load balancing).

cryotek
  • 229
  • 1
  • 4
  • The frontend load balancer, right before nginx, has enabled `externalTrafficPolicy: Local` to preserve client IP. Then the next app's load balancer uses `externalTrafficPolicy: Cluster` with `sessionAffinity: ClientIP` to do sticky session. But it seems most traffic still redirected to one pod. – spondbob Mar 22 '18 at 01:27
  • I understand you are using two LBs set in GKE, one using "externalTrafficPolicy: Local" which has your nginx as backend and another one whcih receives the requests sent by the nginx and is set with "externalTrafficPolicy: Cluster". Am I right? If this is the case and both LBs are Network load balancers, you should try setting both policies to "Local" to preserve the IP in all the path (including the second LB). – cryotek Apr 02 '18 at 15:26
  • Yup you are right. But as per kubernetes doc, the `Local` option "risks potentially imbalanced traffic spreading". That's why I set `Cluster` on 2nd LB. Or is that the drawback of having to enable sticky session where the traffic will be not as balanced as without sticky session? – spondbob Apr 03 '18 at 01:16
  • That is the whole point using session affinity, it will send the traffic to the same backend resource. That is the risk of imbalanced traffic, because everything from the same source IP will be forwarded to the same pod, unless the later is overloaded. – cryotek Apr 03 '18 at 11:40
  • Thanks, I'll have a go. I am accepting this as the answer. – spondbob Apr 04 '18 at 04:54
0

Configure session affinity during the initial configuration of a target pool by setting it to one of the following non-default values in order to provide the desired level of sticky session:

CLIENT_IP

2-tuple hashing, which uses the source and destination IPs. All connections from a client will end up on the same instance regardless of protocol as long as the instance stays healthy.

CLIENT_IP_PROTO

3-tuple hashing, which uses the source and destination IPs and the protocol. All connections from a client will end up on the same instance as long as they use the same protocol and the instance stays healthy.

Lakshman Diwaakar refers to session affinity configuration as being a non-modifiable attribute set "during the creation of" a target pool. His review of Cloud Compute load-balancing functionalities available might be interesting for you to review: Network load balancer Vs HTTP(s) Load Balancer.

remote
  • 31
  • 4
  • Hi thanks for the response. I have checked my target pools are already set to use `CLIENT_IP`, we do that when creating a service of LoadBalancer and set the sessionAffinity to ClientIP. – spondbob Mar 13 '18 at 22:05
  • 1
    I think the problem is the incoming traffic is always assigned to one server only which means it was masked somewhere along the line. – spondbob Mar 13 '18 at 22:05
0

Use the proxy_set_header to add your IP original to NGINX header. Checkout the following examples:

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header X-Forwarded-Host $remote_addr;

proxy_set_header X-Forwarded-For $remote_addr;

In the same way, you can add your IP origin to NGINX header.