TLS handshake fails intermittently when using HAProxy Ingress Controller

Question

I'm using HAProxy Ingress Controller (https://github.com/helm/charts/tree/master/incubator/haproxy-ingress) for TLS-termination for my app. I have a simple Node.JS server listening on 8080 for HTTP, and 1935 as a simple echo server (not HTTP).
And I use HAProxy Ingress controller to wrap the ports in TLS. (8080 -> 443 (HTTPS), 1935 -> 1936 (TCP + TLS))
I installed HAProxy Ingress Controller with

helm upgrade --install haproxy-ingress incubator/haproxy-ingress \
   --namespace test \
  -f ./haproxy-ingress-values.yaml \
  --version v0.0.27

, where the content of haproxy-ingress-values.yaml is


controller:
  ingressClass: haproxy
  replicaCount: 1
  service:
    type: LoadBalancer
  tcp:
    1936: "test/simple-server:1935:::test/ingress-cert"
  nodeSelector:
    "kubernetes.io/os": linux
defaultBackend:
  nodeSelector:
    "kubernetes.io/os": linux

And here's my ingress:


apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: "haproxy"
spec:
  tls:
  - hosts:
    secretName: ingress-cert
  rules:
    - http:
        paths:
        - path: /
          backend:
            serviceName: "simple-server"
            servicePort: 8080

The cert is self-signed. If I test the TLS handshake with

echo | openssl s_client -connect "<IP>":1936

Sometimes (about 1/3 of the times) it fails with

CONNECTED(00000005)
139828847829440:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:332:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 316 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

The same problem doesn't happen for 443 port.
See here for the details of the settings to reproduce the problem.

[edit]
As pointed out by @JoaoMorais, it's because the default statistic port is 1936. Although I didn't turn on statistics, it seems like it still interferes with the behavior.
There're two solutions that work for me.

Change my service's 1936 port to another
Change the stats port by adding values like below when installing the haproxy-ingress chart.

controller:
  stats:
    port: 5000

port `1936` is already used for stats, can you reproduce the problem with another external/listening port number? — Joao Morais, Sep 29 '20 at 10:21
@JoaoMorais Thank you. Changing 1936 port to other port or changing the stats port work! Please reply to this question so that I can mark your answer as correct. :) — someone, Sep 29 '20 at 11:27

score 2 · Accepted Answer · answered Sep 29 '20 at 12:03

HAProxy by default allows to reuse the same port number across the same or other frontend/listen sections and also across other haproxy process. This can be changed adding noreuseport in the global section.

The default HAProxy Ingress configuration uses port number 1936 to expose stats. If such port number is reused by eg a tcp proxy, the incoming requests will be distributed between both frontends - sometimes your service will be called, sometimes the stats page. Changing the tcp proxy or the stats page (doc here) to another port should solve the issue.

TLS handshake fails intermittently when using HAProxy Ingress Controller

1 Answers1