HAProxy as reverse proxy for AWS API Gateway

Question

As the title suggest, I have an AWS API Gateway endpoint that I want to put behind HAProxy.

This is my current HAProxy configuration

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

listen  http
        bind 127.0.0.1:8080
        maxconn     18000

        acl api_gateway path_beg /api-gateway
        use_backend api-gateway-backend if api-gateway

backend api-gateway-backend
        http-request set-header Host xxxxx.execute-api.ap-southeast-2.amazonaws.com
        server api-gateway xxxxx.execute-api.ap-southeast-2.amazonaws.com:443

When I hit the /api-gateway endpoint on my HAProxy, I get 400 Bad Request. See below:

I tried to change the backend to use this server api-gateway xxxxx.execute-api.ap-southeast-2.amazonaws.com:443 ssl verify none but I got 503 Service Unavailable instead.

I think this could be related to SSL SNI configuration that I need to enable on HAProxy, see this forum post https://forums.aws.amazon.com/thread.jspa?threadID=240197

Michael - sqlbot · Accepted Answer · 2019-03-19T23:36:30.713

If you don't use HTTPS then CloudFront will return the 400 Bad Request¹ error because API Gateway doesn't support HTTP.

Adding ssl verify none enables HTTPS to the back-end, CloudFront just closes the connection, causing HAProxy to log the session state at disconnection as SC-- and return the local 503 Service Unavailable error.

The solution is indeed sending Server Name Identification (SNI). Otherwise, CloudFront's front-end doesn't know which SSL certificate to offer you -- the generic *.cloudfront.net wildcard cert, or the one for *.execute-api.ap-southeast-2.amazonaws.com or one of probably hundreds of thousands of other possibilities. That's what SNI does in this case -- it tells the server what name you're connecting to.

The solution -- this is a single line, shown as multiple lines for clarity:

server api-gateway 
       xxxxx.execute-api.ap-southeast-2.amazonaws.com:443
       ssl
       verify none
       sni str(xxxxx.execute-api.ap-southeast-2.amazonaws.com)

You have to use the str() string sample fetch here, because the sni server keyword expects a sample fetch expression (for cases where you wanted to use the incoming request's SNI, for example).

The problem in the question on the forum was not actually related to SNI -- that part of the configuration was correct. The problem there was that they failed to do what you already did correctly -- http-request set-header host ... so that CloudFront sees the correct Host: header.

Note also the advice on the forum that this is a "very bad idea" seems distinctly misplaced.

¹ 400 CloudFront -- in some cases -- returns "Bad Request" in the body but the actual HTTP status code is 403 which corresponds to Forbidden.

Thanks for the solution Michael, that fixed the issue. Yes, I saw that advice as well. The reason behind this is because we are building our own tracking solution using Amazon Lambda and API Gateway. We need the tracker to be under the same domain as our website. — Rudy Lee, Feb 03 '17 at 03:54
@michael-sqlbot haproxy 1.5 doesn't support the ssi instruction, do you know if is there an alternative rather than upgrading haproxy? I mean, is it possible to fix this problem for that version? — Rowinson Gallego, Nov 10 '17 at 21:34
@RowinsonGallego I don't think it is possible in 1.5. However, 1.5 to the latest 1.6 should be a relatively easy and safe migration. — Michael - sqlbot, Nov 10 '17 at 23:00

score 0 · Answer 2 · answered Jul 13 '20 at 22:38

server api-gateway 
       xxxxx.execute-api.ap-southeast-2.amazonaws.com:443
       ssl
       verify none
       sni str(xxxxx.execute-api.ap-southeast-2.amazonaws.com)

For those in the future: You may wish to remove that :443, since at least 1.8, ssl is implied, and if you use the keywords SSL and or SNI with that port defined, you will get the error messages that might lead you down the path of re-compiling.

I got:

unknown keywords for ssl and the same for sni and I couldn't figure out why until I re-read: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.2 more closely.

The error message does lead you to believe that haproxy was not compiled with SSL support, but haproxy -vv will show you if it is.

To be fair, if you didn't compile with USE_OPENSSL=1 and or have a non-standard distro, use SSL_INC=/usr/include/openssl/ (replace path) SSL_LIB=/usr/lib64/openssl (replace path)

if haproxy -vv shows not built for ssl.

HAProxy as reverse proxy for AWS API Gateway

2 Answers2

Linked