I have created an AWS EKS cluster to which I deployed HAPRoxy Ingress Controller backed by a Network Load Balancer (NLB) which is deployed in 3 AZs of the cluster region.
To ensure high availability across multiple Availability Zones in the cluster region, I want to deploy at least one HAProxy pod per AZ.
Another requirement for my application has brought me to use stick tables with the HAPRoxy. However, the stick tables data is not shared among all HAPRoxy pods. The only solution to make the stick table data propagate among all HAProxy pods is by manually adding the IPs and ports of the HAProxy servers to the peers list which is then added to the stick table configurations. But since our environment is highly dynamic and the IPs are not static we won't be able to achieve that. Consequently, without the stick table data propagation, what we believe is happening, is that the client connects to an NLB instance, which then connects to one of the HAPRoxy pods. An entry is then added to the HAPRoxy stick table, on that specific pod. Later on, when the user continues working on the app, the connection from the NLB might go to a different HAPRoxy pod which doesn't contain the session details that were entered to the first stick table. Hence the application will error out that the session doesn't exist on that server.
Thus, the only possible solution that we can think about, is to persist a connection from the client -> NLB -> One HAProxy pod so that the sticky data is constantly hitting the same stick table.
We've looked into enabling stickiness on the NLB targets, but that didn't work, we got the same result. https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#sticky-sessions
Or could it be that the user is getting a connection to a different NLB instances (in a different AZ)?
Is there any other way? Or have we misunderstood certain concepts?
