1

I got OutOfMemoryError when I tried to download a large file (> 10GB) with JDK HttpClient. This only happened when I used BodyHandlers.ofInputStream. It didn't throw any error if I use BodyHandlers.ofFile. I am using JDK 11. Does anybody know how to solve this? I need the InputStream for process.

OutOfMemoryError is thrown:

HttpResponse<InputStream> httpResponse = httpClient.send(request, HttpResponse.BodyHandlers.ofInputStream());
try (InputStream responseStream = httpResponse.body()) {
    Files.copy(responseStream, outputFile);
}

No error is thrown:

HttpResponse<Path> httpResponse = httpClient.send(request, HttpResponse.BodyHandlers.ofFile(outputFile));

Here is the stack trace.

java.io.IOException: closed
    at jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.current(ResponseSubscribers.java:342) ~[java.net.http:?]
    at jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.read(ResponseSubscribers.java:393) ~[java.net.http:?]
    at java.io.InputStream.transferTo(InputStream.java:704) ~[?:?]
    at java.nio.file.Files.copy(Files.java:3078) ~[?:?]
    at <Masked for privacy>
    at <Masked for privacy>
    at <Masked for privacy>
Caused by: java.lang.OutOfMemoryError: Java heap space
    at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:937) ~[?:?]
    at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491) ~[?:?]
    at javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:779) ~[?:?]
    at javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730) ~[?:?]
    at javax.crypto.Cipher.doFinal(Cipher.java:2497) ~[?:?]
    at sun.security.ssl.SSLCipher$T12GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1629) ~[?:?]
    at sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240) ~[?:?]
    at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197) ~[?:?]
    at sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160) ~[?:?]
    at sun.security.ssl.SSLTransport.decode(SSLTransport.java:110) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) ~[?:?]
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) ~[?:?]
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637) ~[?:?]
    at jdk.internal.net.http.common.SSLFlowDelegate$Reader.unwrapBuffer(SSLFlowDelegate.java:481) ~[java.net.http:?]
    at jdk.internal.net.http.common.SSLFlowDelegate$Reader.processData(SSLFlowDelegate.java:392) ~[java.net.http:?]
    at jdk.internal.net.http.common.SSLFlowDelegate$Reader$ReaderDownstreamPusher.run(SSLFlowDelegate.java:264) ~[java.net.http:?]
    at jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175) ~[java.net.http:?]
    at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147) ~[java.net.http:?]
    at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198) ~[java.net.http:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:834) ~[?:?]
daniel
  • 2,665
  • 1
  • 8
  • 18
Franz Wong
  • 1,024
  • 1
  • 10
  • 32
  • it would be better if you download the entire response in a file, split it in chunks and then process those chunks using input stream. – pratap Mar 31 '21 at 09:32
  • What did the heap dump tell you? (How large was the heap? What consumed the memory?) – meriton Mar 31 '21 at 09:33
  • I'd have to guess since I don't know the details of those classes but I assume that the handler returned by `ofFile()` writes the data to disk (as per the JavaDoc) while the `ofInputStream()` handler keeps the data in memory. If so that would explain the OOM you get. As pratap suggested, write the data to a file first and then read it from there - either in chunks or by using a memory-mapped files. – Thomas Mar 31 '21 at 09:46
  • @meriton Files.copy only uses 8KB as buffer, so it should be somewhere else caching the data. I guess it should be related to crypto classes. – Franz Wong Mar 31 '21 at 09:55
  • @pratap That is the last resort if I can't solve it. – Franz Wong Mar 31 '21 at 09:56
  • The `BodyHandler` returned by `ofInputStream()` will keep some of the data in memory, but it shouldn't keep the full 10Gb... What version (java --version) of JDK 11 are you using? Does the problem reproduces on more recent version (JDK 16)? – daniel Mar 31 '21 at 10:19
  • Also do you know whether HTTP/1.1 or HTTP/2 were being used? – daniel Mar 31 '21 at 10:26
  • @daniel I tried with Java 16 and the error didn't occur. It uses HTTP/1.1. Anyway I will save the whole file to disk first with `ofFile`. But I guess `fromSubscriber(Flow.Subscriber super List>)` may solve the problem because `ofFile` is using that. However, I don't have enough time to try that now. – Franz Wong Apr 02 '21 at 02:43
  • 1
    I was wrong. Same problem occurs in Java 16. – Franz Wong Apr 02 '21 at 07:20
  • This is strange. It could be useful to get a heap dump and see what resources are being buffered and from where. – daniel Apr 02 '21 at 14:57

0 Answers0