Java Jersey client - upload 1Gb file with 256Mb client JVM heap

Question

I'm using Jersey 2.28 and want to write a client to upload 1Gb files, while the client JVM heap cannot go above 256Mb.

What are my options here?

I've tried with the snippet below but getting OOM as it seems Jersey keeps reading as much as it can from the InputStream. Can Jersey be instructed to flush in order to deal with large inputs?

public Response upload(InputStream inputStream) {
    Client client = ClientBuilder.newBuilder().build();
    client.register(MultiPartFeature.class);
    WebTarget target = client.target("http://myexample.com").path("/upload");
    Invocation.Builder builder = target.request();

    FormDataMultiPart form = new FormDataMultiPart();
    form.bodyPart(new StreamDataBodyPart("file", inputStream, "filename.zip"));

    return builder.post(Entity.entity(form, form.getMediaType()));
}

Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3236)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    at sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
    at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:200)
    at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:276)

score 1 · Answer 1 · edited Apr 04 '20 at 22:10

The solution is to add:

client.property(ClientProperties.REQUEST_ENTITY_PROCESSING,
                RequestEntityProcessing.CHUNKED);

ClientProperties.REQUEST_ENTITY_PROCESSING:

The property specifies how the entity should be serialized to the output stream by the Connector; if buffering should be used or the entity is streamed in chunked encoding.

The value MUST be an instance of String or an enum value RequestEntityProcessing in the case of programmatic definition of the property. Allowed values are:

BUFFERED: the entity will be buffered and content length will be send in Content-length header.

CHUNKED: chunked encoding will be used and the entity will be streamed.

The default value is CHUNKED. However, due to limitations, some Connectors can define a different default value (usually if the chunked encoding cannot be properly supported on the Connector). This detail should be specified in the Javadoc of that particular Connector. For example, HttpUrlConnector (the default Connector) uses buffering as the default mode.

The String value of the configuration property constant is "jersey.config.client.request.entity.processing".

Java Jersey client - upload 1Gb file with 256Mb client JVM heap

1 Answers1