I currently have the need to read HTTP headers as they are sent. I have tried the new Java 11 HTTP client (java.net.http.HttpClient), but I have no luck intercepting the headers, as they are not available until the content is starting to be sent (so all headers are returned at once).
I need this functionality to handle progress/status updates as the server sends multiple headers with the progress of the operation, and then the response is sent normally once ready. This is not my design (the response is sent from ClickHouse), so I just need to be able to find an HTTP client that can read it.
The curl command line works just fine:
curl -vsS "http://localhost:8123?send_progress_in_http_headers=1&wait_end_of_query=1" -d "select count() from numbers(100000000000) format JSON;"
* Trying 127.0.0.1:8123...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8123 (#0)
> POST /?send_progress_in_http_headers=1&wait_end_of_query=1 HTTP/1.1
> Host: localhost:8123
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Length: 54
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 54 out of 54 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Sat, 20 Feb 2021 11:35:14 GMT
< Connection: Keep-Alive
< Content-Type: application/json; charset=UTF-8
< X-ClickHouse-Server-Display-Name: 78b40ec2d4a2
< Transfer-Encoding: chunked
< X-ClickHouse-Query-Id: 19bd4955-09ad-47f2-a64f-93d926383f55
< X-ClickHouse-Format: JSON
< X-ClickHouse-Timezone: UTC
< Keep-Alive: timeout=3
< X-ClickHouse-Progress: {"read_rows":"100794368","read_bytes":"806354944","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"249954304","read_bytes":"1999634432","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"414515200","read_bytes":"3316121600","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"558759936","read_bytes":"4470079488","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"722206720","read_bytes":"5777653760","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"901185536","read_bytes":"7209484288","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"1110573056","read_bytes":"8884584448","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"1323302912","read_bytes":"10586423296","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"1534722048","read_bytes":"12277776384","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"1729953792","read_bytes":"13839630336","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"1943535616","read_bytes":"15548284928","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"2165768192","read_bytes":"17326145536","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"2382692352","read_bytes":"19061538816","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"2597519360","read_bytes":"20780154880","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"2807758848","read_bytes":"22462070784","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"3021733888","read_bytes":"24173871104","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"3234922496","read_bytes":"25879379968","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"3457875968","read_bytes":"27663007744","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
< X-ClickHouse-Progress: {"read_rows":"3663790080","read_bytes":"29310320640","written_rows":"0","written_bytes":"0","total_rows_to_read":"100000000000"}
I can see from the debug logs for the client that the data is read and parsed incrementally, but there is no way to access the information that I'm aware of:
...
2021-02-21 10:31:03,070 DEBUG [HttpClient-1-SelectorManager] [jdk.internal.httpclient.debug:286] - [HttpClient-1-SelectorManager] [747ms] SelectorAttachment Registering jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent@7e37cce4 for 0 (false)
2021-02-21 10:31:03,072 DEBUG [HttpClient-1-SelectorManager] [jdk.internal.httpclient.debug:286] - [HttpClient-1-SelectorManager] [749ms] SocketTube(1) read bytes: 490
2021-02-21 10:31:03,072 DEBUG [HttpClient-1-SelectorManager] [jdk.internal.httpclient.debug:286] - [HttpClient-1-SelectorManager] [750ms] Http1AsyncReceiver(SocketTube(1)) Putting 490 bytes into the queue
2021-02-21 10:31:03,073 DEBUG [HttpClient-1-SelectorManager] [jdk.internal.httpclient.debug:286] - [HttpClient-1-SelectorManager] [750ms] SocketTube(1) resuming read event
2021-02-21 10:31:03,073 DEBUG [HttpClient-1-Worker-0] [jdk.internal.httpclient.debug:286] - [HttpClient-1-Worker-0] [750ms] Http1AsyncReceiver(SocketTube(1)) Got 490 bytes for delegate jdk.internal.net.http.Http1Response$HeadersReader@2bcbc994
2021-02-21 10:31:03,073 DEBUG [HttpClient-1-SelectorManager] [jdk.internal.httpclient.debug:286] - [HttpClient-1-SelectorManager] [750ms] SelectorAttachment Registering jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent@7e37cce4 for 1 (false)
2021-02-21 10:31:03,073 DEBUG [HttpClient-1-Worker-0] [jdk.internal.httpclient.debug:286] - [HttpClient-1-Worker-0] [750ms] Http1AsyncReceiver(SocketTube(1)) downstream subscription demand is 1
2021-02-21 10:31:03,073 DEBUG [HttpClient-1-SelectorManager] [jdk.internal.httpclient.debug:286] - [HttpClient-1-SelectorManager] [750ms] SocketTube(1) leaving read() loop after onNext: Reading: [ops=1, demand=0, stopped=false], Writing: [ops=0, demand=1]
....
Any ideas on how the information can be extracted as it is becoming available? There seem to be multiple async HTTP clients, but they are not having a callback for the headers.
Edit: I shared my project here: https://github.com/ethlo/clackshack
The issue in question would have to be handled here: https://github.com/ethlo/clackshack/blob/main/src/main/java/com/ethlo/clackshack/Java11Client.java
The project contains a test with a simple HTTP server, responding with HTTP headers with 500 ms delay, to avoid the dependency on ClickHouse for reproducing.
