0

We have a Wildfly instance with a simple java rest application running in it on a 36-core server. The application has no default jobs running in the background, it simply respondes to user requests (< 500 per day).

Our admin noticed a strange behaviour today night. The wildfly instance used apparently the full capacity of the server since 2:00 a.m. At that time no user could have used the application. No log files between the last user action an evening before and the first user action this morning are available.

I've launched the remote debugger in eclipse and would like to understand what all the threads started by Wildfly actually do?

  1. Thread[MSC service thread 1-xx](Running) 68 entries. What are those threads for? Can they have influenced the performance / capacity coverage? Can I restrict the number of those threads? Should I do it?
  2. Deamon Thread [weld-worker-xx](Running) 25 entries.
  3. Thread[default task-x](Running) 8 entries. These are probably the actual user-tasks?
  4. Thread[XNIO-1 I/O-x](Running) 2 entries. Input/Output to the database? Or any file, like logging e.q.?
  5. Daemon Thread [Transaction Reaper](Running)
  6. Deamon Thread [Transaction Reaper Worker 0](Running)
  7. Thread [Periodic Recovery](Running)
  8. A lot of <not responding>. What does that actually mean? The threads are running, but I'm not permittet to have a look in?
Ira Re
  • 730
  • 3
  • 9
  • 25
  • This is quite a broad question, but I suspect that your problems are with the threads that aren't responding. I'd imagine they're the ones that are spinning hard and eating your CPU. – Kayaman May 12 '16 at 08:31
  • You might also want to issue `kill -3 ` to have Wildfly print a thread dump to stdout (which should be redirected to some file). Save the dump for later analysis. – Kayaman May 12 '16 at 08:38
  • Hello Kayaman, thank you for the hint. The `` threads disappeared after a restart and the performance got reasonably better. We'll try to print a thread dump next time the problem occures. I'm still interested in explanations to all the other thread stack entries, just to understand what is usally going on. So any explanations would be great! – Ira Re May 12 '16 at 09:54
  • Well of course they disappeared. That's why you need to gather all possible information **before** restarting. As for the other threads, you've got Wildfly's threads as well as threads potentially belonging to any other libraries you're using. It's too broad to get into here, but Wildfly's documentation and some Googling should clear up a lot of them. If you've managed to attach a debugger to the server, you can see what the threads are doing (or what they're waiting for) and try to deduce their job from that. – Kayaman May 12 '16 at 10:03

1 Answers1

0

In general, you might have a look at the threads subsystem in wildfly:

https://docs.jboss.org/author/display/WFLY8/Threads+subsystem+configuration

This performance tuning guide is also very useful to understand threads and thread pools:

http://www.mastertheboss.com/jboss-server/jboss-performance/wildfly-performance-tuning

The thread types you mention are, AFAICT:

MSC threads are the modular service container threads. They can be limited via the configuration here: https://issues.jboss.org/browse/MSC-144 (-Dorg.jboss.server.bootstrap.maxThreads)

Default threads: the I/O threads in the pools defined by the threads subsystems for various tasks.

XNIO threads: Undertow relies on the XNIO API for creating Worker threads using bounded-queue-thread-pool. https://developer.jboss.org/thread/241230?start=0&tstart=0 - some good performance tuning tips here: http://www.mastertheboss.com/jboss-server/jboss-performance/wildfly-performance-tuning

weld-worker threads are the threads the CDI implementation uses and can be configured as such: http://docs.jboss.org/weld/reference/latest/en-US/html/configure.html#_thread_pool_configuration . It seems weld keeps a thread pool for each deployment (https://issues.jboss.org/browse/WFLY-4653) and is configurable from WF9 onwards in standalone.xml

Transaction Reaper and worker are responsible for monitoring JTA transactions and timeouts etc. http://www.mastertheboss.com/jboss-server/jboss-configuration/configuring-transactions-jta-using-jboss-as7-wildfly http://www.hhutzler.de/blog/a-deeper-dive-into-transaction-timeouts-with-jee7wildfly-and-oracle-rac/

Reriodic recovery thread: After a system crash Periodic Recovery Thread should use XA recover and XA commit to commit this transaction without any user intervention. http://www.hhutzler.de/blog/a-deep-dive-into-2-phase-commit-with-wildfly-and-oracle-rac/

sprockets
  • 981
  • 1
  • 6
  • 16