Using http2_push with nginX in conjunction with HAProxy does not work

Question

I can not get HTTP/2 push to work when nginX is configured behind HaProxy. It does work when nginX is hit directly by the web browser however.

Did a lot of research already but did not find any hints. Hope anybody knows what I am doing wrong. See configuration and further observations below.

Configuration

The relevant HaProxy (version 1.8.7) configuration is given by:


    frontend appname
        bind *:443 ssl crt certificate.pem alpn h2,http/1.1
        mode tcp
        use_backend app-http2 if { ssl_fc_alpn -i h2 }
        default_backend app

    backend app-http2
        mode tcp
        server lamp2 127.0.0.1:8002 check send-proxy

And the relevant nginX (version 1.14.0) configuration is as follows:


    http {
        # This is the one I would like to use
        server {
            listen       8002 http2 proxy_protocol;

            server_name  _;
            root         /usr/share/nginx/html;

            location / {
                http2_push /image.jpg;
            }
        }

        # This one can be accessed directly; and *does* work
        server {
            listen       8004 http2 ssl;

            ssl_certificate certificate.pem;
            ssl_certificate_key private.key;

            server_name  _;
            root         /usr/share/nginx/html;

            location / {
                http2_push /image.jpg;
            }
        }
    }

Observations

In the nginx logs I can verify that both ways of accessing the content do use HTTP2.
When I use Chrome to access the page I can see that push is used only when directly visiting nginX

Update 9 may 2018 Still not solved. But people seem to agree that it is a bug. I opened an issue at their issue tracker: https://trac.nginx.org/nginx/ticket/1549#ticket

Update 26 april 2018

It appears the problem is bigger than just the http2 push. If I log the $scheme nginX variable it is always set to http. Both when accessing from http as from http2.

So that obviously seems like the problem. However I am not sure how I can fix this. Haproxy is working tcp mode; thus will likely not be doing anything wrong.

A related (but possibly outdated) Stack Overflow topic is nginx $scheme variable behind load balancer. But that answer does not help solving this problem!

Update 25 april 2018

Still not working. But a step closer. Ran nghttp2 on both, and results are found below.

Both seem to have the /image.jpg resource embedded. But the one that goes via haproxy has it's scheme set to http; and not to https. As one can see in this diff:

I assume due to this; Chrome will not use this pushed resource. I am however not sure what causes this!

Does anyone have a clue?

The complete output of both commands:


    nghttp -nv https://127.0.0.1:8004/

    [  0.001] Connected
    The negotiated protocol: h2
    [  0.003] send SETTINGS frame 
    (niv=2)
    [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
    [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
    [  0.003] send PRIORITY frame 
    (dep_stream_id=0, weight=201, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=0, weight=101, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=0, weight=1, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=7, weight=1, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=3, weight=1, exclusive=0)
    [  0.003] send HEADERS frame 
    ; END_STREAM | END_HEADERS | PRIORITY
    (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
    ; Open new stream
    :method: GET
    :path: /
    :scheme: https
    :authority: 127.0.0.1:8004
    accept: */*
    accept-encoding: gzip, deflate
    user-agent: nghttp2/1.25.0
    [  0.003] recv SETTINGS frame 
    (niv=3)
    [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
    [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65536]
    [SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
    [  0.003] recv WINDOW_UPDATE frame 
    (window_size_increment=2147418112)
    [  0.003] send SETTINGS frame 
    ; ACK
    (niv=0)
    [  0.003] recv SETTINGS frame 
    ; ACK
    (niv=0)
    [  0.003] recv (stream_id=13) :method: GET
    [  0.003] recv (stream_id=13) :path: /image.jpg
    [  0.003] recv (stream_id=13) :scheme: https
    [  0.003] recv (stream_id=13) :authority: 127.0.0.1:8004
    [  0.003] recv (stream_id=13) accept-encoding: gzip, deflate
    [  0.003] recv (stream_id=13) user-agent: nghttp2/1.25.0
    [  0.003] recv PUSH_PROMISE frame 
    ; END_HEADERS
    (padlen=0, promised_stream_id=2)
    [  0.003] recv (stream_id=13) :status: 200
    [  0.003] recv (stream_id=13) server: nginx/1.14.0
    [  0.003] recv (stream_id=13) date: Wed, 25 Apr 2018 15:08:26 GMT
    [  0.003] recv (stream_id=13) content-type: text/html
    [  0.003] recv (stream_id=13) content-length: 638
    [  0.003] recv (stream_id=13) last-modified: Wed, 25 Apr 2018 11:42:58 GMT
    [  0.003] recv (stream_id=13) etag: "5ae069c2-27e"
    [  0.003] recv (stream_id=13) accept-ranges: bytes
    [  0.003] recv HEADERS frame 
    ; END_HEADERS
    (padlen=0)
    ; First response header
    [  0.004] recv DATA frame 
    ; END_STREAM
    [  0.004] recv (stream_id=2) :status: 200
    [  0.004] recv (stream_id=2) server: nginx/1.14.0
    [  0.004] recv (stream_id=2) date: Wed, 25 Apr 2018 15:08:26 GMT
    [  0.004] recv (stream_id=2) content-type: image/jpeg
    [  0.004] recv (stream_id=2) content-length: 182884
    [  0.004] recv (stream_id=2) last-modified: Sat, 18 Jun 2016 15:42:26 GMT
    [  0.004] recv (stream_id=2) etag: "57656be2-2ca64"
    [  0.004] recv (stream_id=2) accept-ranges: bytes
    [  0.004] recv HEADERS frame 
    ; END_HEADERS
    (padlen=0)
    ; First push response header
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] send WINDOW_UPDATE frame 
    (window_size_increment=33248)
    [  0.004] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] send WINDOW_UPDATE frame 
    (window_size_increment=32925)
    [  0.046] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.046] recv DATA frame 
    [  0.046] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.046] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.046] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] recv DATA frame 
    [  0.090] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.090] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.090] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.090] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.134] recv DATA frame 
    [  0.134] recv DATA frame 
    [  0.134] recv DATA frame 
    ; END_STREAM
    [  0.134] send GOAWAY frame 
    (last_stream_id=2, error_code=NO_ERROR(0x00), opaque_data(0)=[])

and


    nghttp -nv https://127.0.0.1:8002/

    [  0.001] Connected
    The negotiated protocol: h2
    [  0.003] send SETTINGS frame 
    (niv=2)
    [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
    [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
    [  0.003] send PRIORITY frame 
    (dep_stream_id=0, weight=201, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=0, weight=101, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=0, weight=1, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=7, weight=1, exclusive=0)
    [  0.003] send PRIORITY frame 
    (dep_stream_id=3, weight=1, exclusive=0)
    [  0.003] send HEADERS frame 
    ; END_STREAM | END_HEADERS | PRIORITY
    (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
    ; Open new stream
    :method: GET
    :path: /
    :scheme: https
    :authority: 127.0.0.1:8002
    accept: */*
    accept-encoding: gzip, deflate
    user-agent: nghttp2/1.25.0
    [  0.003] recv SETTINGS frame 
    (niv=3)
    [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
    [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65536]
    [SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
    [  0.003] recv WINDOW_UPDATE frame 
    (window_size_increment=2147418112)
    [  0.003] send SETTINGS frame 
    ; ACK
    (niv=0)
    [  0.004] recv SETTINGS frame 
    ; ACK
    (niv=0)
    [  0.004] recv (stream_id=13) :method: GET
    [  0.004] recv (stream_id=13) :path: /image.jpg
    [  0.004] recv (stream_id=13) :scheme: http
    [  0.004] recv (stream_id=13) :authority: 127.0.0.1:8002
    [  0.004] recv (stream_id=13) accept-encoding: gzip, deflate
    [  0.004] recv (stream_id=13) user-agent: nghttp2/1.25.0
    [  0.004] recv PUSH_PROMISE frame 
    ; END_HEADERS
    (padlen=0, promised_stream_id=2)
    [  0.004] recv (stream_id=13) :status: 200
    [  0.004] recv (stream_id=13) server: nginx/1.14.0
    [  0.004] recv (stream_id=13) date: Wed, 25 Apr 2018 15:08:45 GMT
    [  0.004] recv (stream_id=13) content-type: text/html
    [  0.004] recv (stream_id=13) content-length: 638
    [  0.004] recv (stream_id=13) last-modified: Wed, 25 Apr 2018 11:42:58 GMT
    [  0.004] recv (stream_id=13) etag: "5ae069c2-27e"
    [  0.004] recv (stream_id=13) accept-ranges: bytes
    [  0.004] recv HEADERS frame 
    ; END_HEADERS
    (padlen=0)
    ; First response header
    [  0.004] recv DATA frame 
    ; END_STREAM
    [  0.004] recv (stream_id=2) :status: 200
    [  0.004] recv (stream_id=2) server: nginx/1.14.0
    [  0.004] recv (stream_id=2) date: Wed, 25 Apr 2018 15:08:45 GMT
    [  0.004] recv (stream_id=2) content-type: image/jpeg
    [  0.004] recv (stream_id=2) content-length: 182884
    [  0.004] recv (stream_id=2) last-modified: Sat, 18 Jun 2016 15:42:26 GMT
    [  0.004] recv (stream_id=2) etag: "57656be2-2ca64"
    [  0.004] recv (stream_id=2) accept-ranges: bytes
    [  0.004] recv HEADERS frame 
    ; END_HEADERS
    (padlen=0)
    ; First push response header
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] send WINDOW_UPDATE frame 
    (window_size_increment=33406)
    [  0.004] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.004] recv DATA frame 
    [  0.044] recv DATA frame 
    [  0.044] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.044] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.044] recv DATA frame 
    [  0.044] recv DATA frame 
    [  0.045] recv DATA frame 
    [  0.045] recv DATA frame 
    [  0.045] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.045] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.045] recv DATA frame 
    [  0.045] recv DATA frame 
    [  0.045] recv DATA frame 
    [  0.045] recv DATA frame 
    [  0.045] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.045] send WINDOW_UPDATE frame 
    (window_size_increment=32767)
    [  0.045] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.046] send WINDOW_UPDATE frame 
    (window_size_increment=32768)
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    [  0.046] recv DATA frame 
    ; END_STREAM
    [  0.046] send GOAWAY frame 
    (last_stream_id=2, error_code=NO_ERROR(0x00), opaque_data(0)=[])

Would you be able to reproduce the issue with `nghttp`? What's the output of `nghttp -nv https://127.0.0.1:8002` vs `nghttp -nv https://127.0.0.1:8004`? What about chrome logs in chrome://net-internals/#http2 ? — Frederik Deweerdt, Apr 25 '18 at 14:10
Thanks Frederick. I ran nghttp and found some interesting results. I updated the topic with these observations. However: now I am not sure what causes this difference! — Roel van Duijnhoven, Apr 25 '18 at 15:17

score 1 · Accepted Answer · answered Jul 05 '18 at 08:03

1

I ended up opening a ticket on the NginX tracker. And it got fixed. The fix is available in the recently released 1.15.1 version.

Bugfix: HTTP/2 server push did not work if SSL was terminated by a proxy server in front of nginx.

Thanks for the help!

answered Jul 05 '18 at 08:03

Roel van Duijnhoven

800
1
7
24

score 0 · Answer 2 · answered Apr 25 '18 at 16:32

0

It appears that Chrome makes sure that if a push is sent on a stream that corresponds to a request served over the https scheme, the promise also uses the same scheme: https://chromium.googlesource.com/chromium/src/+/master/net/spdy/chromium/spdy_session.cc#1766

I'm not sure that the check is correct: the RFC 7540 says that the server sending a PUSH_PROMISE should be authoritative. For http that means that the host name matches, so the browser should be able to process the file.

That said, even if the browser accepted the push, it would only use it if a request against http://127.0.0.1:8002/image.jpg were issued by the browser. If the HTML was obtained via https and it requested /image.jpg, then i'm not sure if the browser would accept to fetch http://127.0.0.1:8002/image.jpg.

Which brings us to nginx setting the scheme as http. I presume this is due to the fact that haproxy does the SSL termination, so ngnix sees a cleartext connection incoming, and as far as it's concerned, the scheme is http. I don't know enough about ngnix to suggest a fix for this.

answered Apr 25 '18 at 16:32

Frederik Deweerdt

4,943
2
29
31

You seem to be spot on. I logged nginX' `$scheme` variable and it is always set to http. Both for http/1.1 as http/2 traffic! I am unable to find similar cases or fixes for this however :(. – Roel van Duijnhoven Apr 26 '18 at 06:07
If using another server is a option, H2O doesn't have this issue: https://github.com/h2o/h2o/blob/master/lib/http2/connection.c#L1484 it copies the scheme from the original stream, as expected. – Frederik Deweerdt Apr 27 '18 at 13:56
Just tested Apache and it similarly works for this (i.e. it copies the scheme from the original request). Also I'm not sure a browser would ever consider a plaintext HTTP/2 pushed resource as authoritative given that they don't support HTTP/2 over HTTP? – Barry Pollard Apr 29 '18 at 09:36
I can't say that browsers will ever support that but there are efforts to define using an `http` scheme over a TLS HTTP/2 connection: https://tools.ietf.org/html/rfc8164#section-2 -- please note that the scheme and the use of TLS are two different things. So you can, in theory, use and `http` scheme over a TLS connection. – Frederik Deweerdt Apr 29 '18 at 14:22
I would think that using SSL termination in combination with a load balancer would be a very common situation (HaProxy, nginX and Apache make up a large part of the ecosystem). So: how are others using that software doing it? Re-encoding the SSL traffic with an easy to encode certificate? Is the scheme actually available in the HTTP request? It is not in the the proxy protocol spec as far as I can see (for example http://rve.org.uk/dumprequest). And it is also not in the proxy protocol spec. If that is the case; that would explain why it is isn't that easy for both pieces of software. – Roel van Duijnhoven May 01 '18 at 06:31
It is a common scenario, but i suspect HTTP/2 pushes are uncommon enough that ngnix has this bug. The scheme is part of the request in HTTP, it has bee explicitly made part of the request in HTTP/2, see the second bullet point here: https://tools.ietf.org/html/rfc7540#section-8.1.2.3 – Frederik Deweerdt May 01 '18 at 15:32
1

I opened an issue at their issue tracker: https://trac.nginx.org/nginx/ticket/1549#ticket – Roel van Duijnhoven May 09 '18 at 14:36

Using http2_push with nginX in conjunction with HAProxy does not work

2 Answers2