3

I've a Java + Spring app that will query ElasticSearch using Jest client (poor choice because it is poorly documented). ElasticSearch has response times of about 8-20 ms with 150 concurrent connections, but my app goes up to 900 -1500 ms. A quick look at VisualVM tells me that the processor usage is below 10% and profiling it tells me that 98% of the time all that the app does is wait on the following method

org.apache.http.pool.PoolEntryFuture.await()

that is part of Apache HttpCore and a dependency of Jest. I don't have a limitation in terms of threads that can run on tomcat (max is 200 and VisualVM says that the maximum number of thread during the experiment was 174). So it's not waiting free threads.

I think that the latency increase is excessive and I suspect that Jest is using an internal threadpool that has not enough threads to cope with all the requests, but I don't know.

Thoughts?

Oleg Valter is with Ukraine
  • 9,619
  • 8
  • 37
  • 59
gotch4
  • 13,093
  • 29
  • 107
  • 170

2 Answers2

4

I think that the latency increase is excessive and I suspect that Jest is using an internal threadpool that has not enough threads to cope with all the requests...

In poking around the source real fast I see that you should be able to inject a ClientConfig into the Jest client factory.

The ClientConfig has the following setters which seem to impact the internal Apache http client connection manager:

clientConfig.maxTotalConnection(...);
clientConfig.defaultMaxTotalConnectionPerRoute(...);
clientConfig.maxTotalConnectionPerRoute(...);

Maybe tweaking some of those will give you more connections? Take a look at the JestClientFactory source to see what it is doing. We've definitely had to tweak those values in the past when making a large number of connections to the same server using HttpClient.

Gray
  • 115,027
  • 24
  • 293
  • 354
  • yup, I found that too. There were not enough connections available. I set this to the same number of threads for tomcat and it started running fast. I had to browse the sources on github though. A little bit of documentation for Jest wouldn't be bad. Shall we consider adding a Jest tag? – gotch4 Oct 22 '13 at 20:34
  • Being an author of open software (which has some decent documentation), I always respond to folks that complain about missing docs with "feel free to write them and submit them". So I guess I need to keep my mouth shut here @gotch4. Best of luck. :-) – Gray Oct 22 '13 at 20:37
0

I would test this with just one connection and see what the average response time is. With just one thread you should have more than enough thread and resources etc. Most likely the process is waiting on an external resource like a database or a network service.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130