Jboss application running on 3 datacenters, and when routed to handle entire traffic on single datacenter we got errors saying 'Rejected command because thread-pool queueSize at rejection threshold'. And, CPU utilization were 100% in datacenter.
Existing thread pool configuration: maximumSize=30 maxQueueSuze=30 queueSizeRejectionThreshold=30
Application peak TPS after issue : healthy state: 1700 per Datacenter 28 PODs configured: 58 TPS per POD is handled at few pods at 15% utilization. Application response time, average: 46ms, p99: 270ms.
Day of issue- Application peak TPS on single datacenter: 4234 per pod TPS: 235 (configured at 18 pods) CPU - 100% utilization
Questions are, On thread pool configuration for hystrix formula is requests per second * p99 response + breathing room. Should I take peak TPS of 235 when traffic was on single datacenter?
Which is correct way to configure thread pool to handle load on single datacenter? And, is there a formula I could use to configure capacity / number of pods to handle load on single datacenter based on the stats?
