3

We're using Wildfly 10.1.0 and Resteasy 3.1.1. At throughput in the hundreds of rps we see randomly long requests even though the average latency is very low.

We are looking at metrics generated from New Relic on our application. We notice that a small percentage of requests, ~1% get stuck inside "HttpServlet30Dispatcher.service()" which is a Resteasy lib function. For the requests that get stuck there, it looks like HttpServlet30Dispatcher (or some code underneath it) consumes 100% of the time of that request. We see requests that normally take 50ms go up to 20s+.

Does anyone know what could cause this? It feels like Wildfly could be blocking the request waiting for a free thread, and maybe thread starvation happens. Can anyone confirm, or any advice on how to profile this?

Thanks! Matt

Matthew James
  • 63
  • 7
  • I have the exact same behaviour, and the exact same versions. And i just landed here. if i find something i will post as well here. And as well new relic,... i've a quite high range of thread max configured on the subsystem for the workers, and this happen as well at night where almost no user is connected. Btw Are you using web sockets ? – Tama Dec 06 '18 at 22:24
  • I think it is slow mobile devices. I can reproduce the issue by slowing down network speeds in the Chrom dev tools and then making a request. – Matthew James Dec 08 '18 at 02:29
  • @MatthewJames were you able to find the solution for this problem? – Ruslan Sep 05 '19 at 18:07
  • Hi Yes I’ll post it below – Matthew James Sep 07 '19 at 07:28

1 Answers1

2

The answer is that these are caused by slow network. I was able to prove this by using the chrome dev tools and setting the network to have very slow downloads.

What’s happening is: Wildfly is the place where responses are buffered, so a slow network client will cause this function to block.

New Relic is the tool we’re using and it isn’t able to filter out these requests, so they add to our overall APDEX score which is annoying.

This actually doesn’t indicate any problem in your code or app server configuration. Just note that slow network clients will chew up a thread while this happens. I have been able to reproduce the exact same issue in NodeJS and I imagine it’s the same in other app servers. I would’ve though Wildfly is using NIO and it would make this transaction async but from what I can tell that’s not happening.

There are plenty of solutions to IO buffering - you can use NGINX or some other proxy to do the buffering for you and your app server will not have to do it.

Renan Ivo
  • 1,368
  • 9
  • 17
Matthew James
  • 63
  • 7