Here's what I'm trying to do: I have a bunch of hosts running reverse proxied MinIO instances (using nginx) and I want to load balance requests by using HAProxy. I don't want to use TLS termination and instead terminate at the targeted hosts.
I used Sidekick earlier, but that doesn't support TLS passthrough. The behavior there is however as I would expect: different S3 multipart requests are sent to different hosts.
I've setup HAProxy to load balance between my 12 hosts and ran some tests using balance roundrobin and balance leastconn, but whatever I do my multipart uploads all end up being sent to a single host. Weirdly enough when I use mc ls to list all objects in my S3 backend I can see the requests being routed and logged on the host running haproxy. If I use mc cp however, nothing gets logged -- even when the upload finishes. Activating request tracing in my MinIO instances shows me that all chunks end up being sent to the same host. I would have expected them to be distributed amongst all 12 hosts. AFAIK there is no keep-alive in mc and every chunk request is independent from the last.
It looks like this might be related to the mc commands I send, but I've found some commands (du, rm) which will generate multiple log entries on the haproxy server indicating that different requests for the same command were routed to different servers.
My haproxy is version 2.0.14 and I'm using the following haproxy.cfg:
global
log 127.0.0.1 local0 debug
maxconn 32768
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
defaults
log global
mode tcp
option tcplog
timeout connect 5s
timeout client 50s
timeout server 450s
frontend https
bind :443
default_backend bk_app
backend bk_app
balance roundrobin
option httpchk GET /minio/health/live
option ssl-hello-chk
http-check expect status 200
server fs-testcluster-robert2-n1 10.10.11.150:443 check-ssl verify none
server fs-testcluster-robert2-n4 10.10.11.151:443 check-ssl verify none
server fs-testcluster-robert2-n2 10.10.11.152:443 check-ssl verify none
server fs-testcluster-robert2-n5 10.10.11.153:443 check-ssl verify none
server fs-testcluster-robert2-n3 10.10.11.154:443 check-ssl verify none
server fs-testcluster-robert2-n6 10.10.11.155:443 check-ssl verify none
server fs-testcluster-robert-n1 10.10.12.196:443 check-ssl verify none
server fs-testcluster-robert-n2 10.10.12.198:443 check-ssl verify none
server fs-testcluster-robert-n3 10.10.12.237:443 check-ssl verify none
server fs-testcluster-robert-n4 10.10.11.160:443 check-ssl verify none
server fs-testcluster-robert-n5 10.10.11.161:443 check-ssl verify none
server fs-testcluster-robert-n6 10.10.11.162:443 check-ssl verify none
Can anybody explain what's going on or what I might have misconfigured?
