Does Jetty's httpClient.setResponseBufferSize() method do anything?

Question

I'm building a simple proxy server using Jetty's HttpClient. I am using Jetty version 9.3.10.v20160621 on Java 1.8.0_45.

I have a situation in which I do a GET on a resource which will return a response that is approximately 3.5M in size (I determined this using soapUI). Knowing that Jetty's default max response size is 2M, I do the following when I create the HttpClient instance.

        HttpClient client = new HttpClient();
        client.setResponseBufferSize(4194304);
        client.start();

Later I perform a synchronous GET request like so:

        System.out.println("response buffer size = " + client.getResponseBufferSize());
        retVal = client.GET(uri);

The console log has the following:

response buffer size = 4194304

Nevertheless I get a java.util.concurrent.ExecutionException when I execute the GET() (stack trace below). I can only conclude that either (a) there is a bug in the setResponseBufferSize() method or, (b) the setResponseBufferSize() method doesn't do what the documentation says it does (set the response buffer size). Does anybody know what is up with this?

java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Buffering capacity exceeded
    at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118)
    at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101)
    at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:652)
    at org.eclipse.jetty.client.HttpClient.GET(HttpClient.java:343)
    at oracle.paas.tools.sifter.proxy.ProxySession.get(ProxySession.java:106)
    at oracle.paas.tools.sifter.proxy.Endpoint.doGet(Endpoint.java:75)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:837)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
    at org.eclipse.jetty.server.Server.handle(Server.java:524)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:319)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:253)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Buffering capacity exceeded
    at org.eclipse.jetty.client.util.BufferingResponseListener.onContent(BufferingResponseListener.java:114)
    at org.eclipse.jetty.client.api.Response$Listener$Adapter.onContent(Response.java:248)
    at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:124)
    at org.eclipse.jetty.client.ResponseNotifier.access$100(ResponseNotifier.java:35)
    at org.eclipse.jetty.client.ResponseNotifier$ContentCallback.process(ResponseNotifier.java:272)
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
    at org.eclipse.jetty.client.ResponseNotifier.notifyContent(ResponseNotifier.java:117)
    at org.eclipse.jetty.client.HttpReceiver.responseContent(HttpReceiver.java:326)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.content(HttpReceiverOverHTTP.java:256)
    at org.eclipse.jetty.http.HttpParser.parseContent(HttpParser.java:1584)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1332)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.parse(HttpReceiverOverHTTP.java:158)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:119)
    at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:69)
    at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:90)
    at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:115)
    ... 9 more

@Joakim Erdfelt thanks for that. – gilbertpilz Jul 01 '16 at 15:30 — gilbertpilz, Jul 01 '16 at 15:30

Joakim Erdfelt · Accepted Answer · 2020-07-07T13:48:52.703

HttpClient.setResponseBufferSize(int) is for configuring the internal buffer size used to read individual buffers from the network.

It has no relationship to your Response body content size.

The situation you are dealing with is that the simplistic HttpClient.GET(uri) call will buffer up the response into a FutureResponseListener which is limited to 2MB of memory usage.

This limit is merely a side effect of the java Future concepts.

This is a shorthand / convenience method meant for small responses.

You have a response that is larger, so its on you to use the asynchronous features of HttpClient to read from the response content buffers/streams that HttpClient is managing, processing the data according to what you need it to do.

Example:

    // In Initialization Code, start the client
    HttpClient client = new HttpClient();
    client.start();

    // In code much later on, use the client
    // Don't constantly start/stop the HttpClient
    // Treat the HttpClient as a browser, and each newRequest() as
    //   a tab on that browser.
    InputStreamResponseListener listener = new InputStreamResponseListener();
    // Send asynchronously with the InputStreamResponseListener
    client.newRequest(uri).send(listener);

    // Call to the listener's get() blocks until the headers arrived
    Response response = listener.get(5, TimeUnit.SECONDS);

    // Now check the response information that arrived to decide whether to read the content
    if (response.getStatus() == 200)
    {
        byte[] buffer = new byte[256];
        try (InputStream input = listener.getInputStream())
        {
            while (true)
            {
                int read = input.read(buffer);
                if (read < 0)
                    break;
                // Do something with the bytes just read
            }
        }
    }
    else
    {
        response.abort(new Exception());
    }

Check out the HttpClient Usage.java for more examples

score 5 · Answer 2 · answered Jul 01 '16 at 15:38

5

I found another way of solving my problem. This solution is not as general as Joakim's solution. If the response is greater than 4 MiB (or whatever you specify) it will get the same exception. On the other hand, it's less code.

Request request = client.newRequest(uri);
FutureResponseListener listener = new FutureResponseListener(request, 4 * 1024 * 1024);
request.send(listener);
retVal = listener.get();

answered Jul 01 '16 at 15:38

gilbertpilz

1,708
20
30

That will allocate 4MB for every request, regardless if the response is actually that size or not. – Joakim Erdfelt Jul 07 '16 at 19:56
I know. As I said, your solution is much cleaner and more general. This just got me past that issue so I can move one. – gilbertpilz Jul 07 '16 at 20:27
3

This is a valid way now since buffer will be created on needed based on response.length (https://github.com/eclipse/jetty.project/blob/6b335877d9cc8a6be0c61c5d6d2a559ab9035323/jetty-client/src/main/java/org/eclipse/jetty/client/util/BufferingResponseListener.java#L121-L125) – Cao Manh Dat Oct 25 '18 at 02:30
@CaoManhDat that only applies to responses with specified content length, which a dwindling number of responses actually contain. (Example: HTTP/1.x with closure termination, and Transfer-Encoding chunked) – Joakim Erdfelt Jun 16 '21 at 01:39

Does Jetty's httpClient.setResponseBufferSize() method do anything?

2 Answers2

Linked