0

We have used hystrix - circuit breaker pattern [library] in our one of the module. usecase is:- we are polling 16 number of messages from kafka and processing them using pararllel stream,so, for each message in workflow it takes 3 rest calls which are protected by hystric command. Now, issue is when I try to run my single instance then CPU shows spikes and thread dump shows many threads in waiting state for all the 3 commands. Like below:-

Omitted thread name but assume all all thread pools it shows same thing:-

Thread Pool-7" #82 Thread State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000004cee2312> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Could you please help me in fine tuning application and thread pool parameters? what I am missing here?

Rahul Singh
  • 781
  • 11
  • 27

1 Answers1

0

The default isolation strategy of Hystrix is threadpool and its default size is just 10. It means that only 10 REST calls can be running at the same time in your case.

First, try to increase the below default property to big one.

hystrix.threadpool.default.coreSize=1000  # default is 10

If it works, adjust the value to the proper one. default can be replaced with the proper HystrixThreadPoolKey for each thread pool.

If you are using Semaphore isolation strategy, try to increase the below one.

hystrix.command.default.execution.isolation.semaphore.maxConcurrentRequests=1000

Above one's default is also just 10. default can be replaced with HystrixCommandKey name for each semaphore.

Updated

To choose the isolation strategy, you can use the below property.

hystrix.command.default.execution.isolation.strategy=THREAD or SEMAPHORE

default can be replaced with HystrixCommandKey. It means that you can assign different strategy for each hystrix command.

yongsung.yoon
  • 5,489
  • 28
  • 32
  • okay, thanks for info. So, it means for thread type of strategy, in pool there will connection available depending on core size. Could you please help me in understanding which one to use thread vs semaphore.? – Rahul Singh Feb 07 '19 at 11:09
  • One wrong understanding in your questions is that each command will be executed one thread after acquiring a thread from the pool in case of thread isolation strategy. It's thread pool, not connection pool. So acquiring a connection will be done by the connection pool that your http client is using. It's totally irrelevant to hystrix. – yongsung.yoon Feb 08 '19 at 11:18
  • As for the thread vs semaphore, I've updated my above answer. – yongsung.yoon Feb 08 '19 at 11:19