2

I'm trying stream the data from an HTTP (GET) response to another HTTP (POST) request. With old HttpURLConnection I would take the responses OutputStream, read parts into a buffer and write them to the requests InputStream.

I've already managed to do the same with HttpClient in Java 11 by creating my own Publisher that is used in the POST to write the request body. The GET request has a BodyHandler with ofByteArrayConsumer that sends the chunks to the custom Publisher which itself then sends the chunks to the subscribing HTTP POST request.

But I think this is not the correct approach as it looks like there is something in the API that looks like this could be done directly without implementing publishers and subscribers myself.

There is HttpResponse.BodyHandlers.ofPublisher() which returns a Publisher<List<ByteBuffer> which I can use for the HTTP GET request. Unfortunately for my POST request, there is HttpRequest.BodyPublishers.fromPublisher which expects a Publisher<? extends ByteBuffer> so it seems that the fromPublisher only works for a publisher that holds a complete ByteBuffer and not one that sends several ByteBuffers for parts of the data.

Do I miss something here to be able to connect the BodyPublisher from one request to the other?

Jens
  • 495
  • 1
  • 6
  • 28

1 Answers1

4

You're not missing anything. This is simply a use case that is not supported out of the box for now. Though the mapping from ByteBuffer to List<ByteBuffer> is trivial, the inverse mapping is less so. One easy (if not optimal) way to adapt from one to the other could be to collect all the buffers in the list into a single buffer - possibly combining HttpResponse.BodyHandlers.ofPublisher() with HttpResponse.BodyHandlers.buffering() if you want to control the amount of bytes in each published List<ByteBuffer> that you receive from upstream.

daniel
  • 2,665
  • 1
  • 8
  • 18
  • I've also came to the conclusion that this API is good for streaming into or out of an HTTP stream to other reactive enabled code but it seems not to be designed to be used with another HTTP call. So it is easy to combine Body* (BodyPublisher, etc) with normal * (Publisher, etc) but not not Body* with Body* since they are very special Publishers, etc. – Jens Aug 09 '19 at 07:19
  • I've switched to using "ofInputStream" in both calls and give the response Inputstream to the Requests publisher. This works very smoothly so far, expect two things: I don't find a way to handle any errors, if there happens any exception while stream reading, because the sendAsync future finishes right after processing the headers and there is absolutly silence after that (not even a stacktrace sysout). The other thing is, I don't find a way to check if the call is completly finished (except wrapping the input stream and check that close has been called by HttpClient). – Jens Aug 09 '19 at 07:24
  • Could you elaborate on the error handling? If there is an error reading from the stream then the completable future returned by sendAsync should be completed exceptionally. If not, it's a bug. However be aware that using ofInputStream() introduces a mix of blocking (synchronous) code in the asynchronous code - if the data is not immediately available. that is - when sendAsync tries to send the request body it will block until the requested data is received and handed over to the InputStream. – daniel Aug 09 '19 at 14:12
  • 1
    I think it is not a bug. syndAsync completes as soon as the headers are read. This is also described in the JavaDoc. So I think this bad behaviour is intentionally. You can try it out by wrapping the inputStream and throwing an exception in read(). The future.get() will give you the inputStream and the response object with statucs code 200 and headers is there. Also futures .handle method will go the success way, because it doesnt care what someone is doing with the input stream. So I think there is no chance to handle errors that happen after the HTTP headers. – Jens Aug 14 '19 at 10:11
  • The implementation of `HttpRequest.BodyPublishers.ofInputStream` treats `IOException` as `EOF`. This is at best questionable - it should probably invoke onError() on its subscribers instead. (logged as JDK-8229703) – daniel Aug 14 '19 at 11:18
  • Yes, when debugging I've also seen that there is a method that treats Exceptions and read() == -1 the same way, so there is no chance to catch anything from anywhere because it silently just ends everything as it would when the stream completes. – Jens Aug 14 '19 at 12:48