HAProxy transparent TCP proxy fails when source 0.0.0.0 usesrc clientip is set

Question

I have a setup where I want a transparent TCP proxy in front of some HTTPS services. The services themselves deal with certificates so I'm trying to avoid a HTTP proxy here. I also need the services to get the client's original IP.

The setup is like this:

1. 192.168.86.33: HAProxy
2. 192.168.86.31: Service1
3. 192.168.86.36: Service2 All three run Raspberry Pi OS.

Based on docs it seems like HAProxy is a good tool for the job but I can't get it to work. It works until I add source 0.0.0.0 usesrc clientip but the service sees the IP of the proxy instead of the client.

When I add source 0.0.0.0 usesrc clientip I get this in the client: Error: Client network socket disconnected before secure TLS connection was established And this in the HAProxy log when run with strace:

[{EPOLLIN, {u32=15, u64=15}}], 200, 60000, NULL, 8) = 1
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=304312352}) = 0
read(15, "c", 1024)                     = 1
read(15, 0x7fe9410d50, 1024)            = -1 EAGAIN (Resource temporarily unavailable)
setsockopt(17, SOL_TCP, TCP_NODELAY, [1], 4) = 0
recvfrom(17, 0x55a7d68c80, 15360, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 18
fcntl(18, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
setsockopt(18, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(18, SOL_IP, IP_BIND_ADDRESS_NO_PORT, [1], 4) = 0
setsockopt(18, SOL_IP, IP_TRANSPARENT, [1], 4) = 0
setsockopt(18, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
bind(18, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("192.168.86.161")}, 16) = 0
connect(18, {sa_family=AF_INET, sin_port=htons(30989), sin_addr=inet_addr("192.168.86.36")}, 16) = -1 EINPROGRESS (Operation now in progress)
epoll_ctl(4, EPOLL_CTL_ADD, 17, {EPOLLIN|EPOLLRDHUP, {u32=17, u64=17}}) = 0
epoll_ctl(4, EPOLL_CTL_ADD, 18, {EPOLLIN|EPOLLOUT|EPOLLRDHUP, {u32=18, u64=18}}) = 0
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=307519852}) = 0
epoll_pwait(4, [{EPOLLIN, {u32=17, u64=17}}], 200, 5001, NULL, 8) = 1
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=307766259}) = 0
recvfrom(17, "\26\3\1\2\0\1\0\1\374\3\3\277\357D\241\236\245Gw\361\16\6\273\234\212D\245\305\241.\10o"..., 15360, 0, NULL, NULL) = 517

My full haproxy.cfg below. I have temporarily removed out one of the backends:

defaults
mode tcp
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms

frontend db
   bind 192.168.86.33:30989 transparent 
   default_backend databases

 backend databases
   source 0.0.0.0 usesrc clientip
   server db2 192.168.86.36:30989

Connecting directly to 192.168.86.36 works. Connecting via the proxy works if I remove the "source" line but my service sees the wrong IP.

I believe I have the right kernel modules loaded:

lsmod | grep proxy
nft_tproxy             16384  0
nf_tproxy_ipv6         16384  1 nft_tproxy
nf_tproxy_ipv4         16384  1 nft_tproxy
nf_defrag_ipv6         20480  3 nf_conntrack,xt_socket,nft_tproxy
nf_defrag_ipv4         16384  3 nf_conntrack,xt_socket,nft_tproxy
nf_tables             237568  212 nft_compat,nft_tproxy,nft_chain_nat,nft_socket
ipv6                  557056  50 nf_tproxy_ipv6,bridge,br_netfilter,nf_socket_ipv6

These are the iptables and routes set on the proxy:

#!/bin/bash
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 111
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 111 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

Based on the response here I have also set the proxy as the default gateway on the service machine:

ip route show
default via 192.168.86.33 dev wlan0 src 192.168.86.36 metric 303 

Final things I've tried:

• Recompiled haproxy with USE_LINUX_TPROXY=1: OPTIONS = USE_LINUX_TPROXY=1
• I have set net.ipv4.ip_forward & net.ipv4.ipnonlocalbind to 1

Any help as to where I am going wrong would be much appreciated!