I have a third party application named oVirt, We need to connect to this application using their Rest API exposed.
We have 4 GB RAM in the VM and allocated 1GB for the third party application.
Also this is the CPU configuration,
[root@test ~]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 23
Model name: Pentium(R) Dual-Core CPU E5200 @ 2.50GHz
Stepping: 10
CPU MHz: 1200.000
CPU max MHz: 2500.0000
CPU min MHz: 1200.0000
BogoMIPS: 4999.77
L1d cache: 32K
L1i cache: 32K
L2 cache: 2048K
NUMA node0 CPU(s): 0,1
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl est tm2 ssse3 cx16 xtpr pdcm xsave lahf_lm dtherm
NOTE : The oVirt application itself exposed the JMX support.
Then we start to hit the rest API of the application with 500 number of request in which 30 request are hit parallely. I could see that it scales successfully. I could see that memory is not even using 600 MB during the API hit.
Then we increased the concurrent hits to 32 with 500 request and it failed without any error saying timed out.
I increases the RAM for third party app to 2GB still it fails at 35 concurrent request and sometimes it fails at 32 request again.
I have one more environment with 2 GB Ram running the same environment with different CPU configuration,
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Model name: Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz
Stepping: 9
CPU MHz: 1600.132
BogoMIPS: 6200.36
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
NUMA node0 CPU(s): 0-3
It can pass 63 concurrent request, so what is the point in it? I didnt understand the issue in Scaling.
I observed one log in their application :
2018-03-02 10:58:39,821+05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineScheduled' is using 0 threads out of 100 and 1 tasks are waiting in the queue.
2018-03-02 10:58:39,822+05 INFO [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread pool 'engineThreadMonitoring' is using 1 threads out of 1 and 0 tasks are waiting in the queue.
How to analyse this ?
Could somebody explain the below threading output?
EDITED :
Added JMX response :
[standalone@127.0.0.1:8706 /] ls /core-service=platform-mbean/type=threading
all-thread-ids=[559L,558L,557L,556L,555L,554L,553L,552L,455L,399L,326L,325L,302L,301L,300L,299L,298L,297L,296L,295L,294L,293L,292L,289L,288L,287L,286L,285L,284L,283L,282L,281L,280L,279L,278L,277L,276L,275L,274L,273L,272L,271L,269L,264L,263L,262L,261L,260L,259L,258L,257L,256L,255L,254L,252L,251L,242L,237L,236L,235L,234L,233L,232L,231L,230L,227L,226L,225L,224L,223L,222L,221L,220L,219L,218L,217L,216L,215L,214L,213L,212L,211L,209L,208L,206L,205L,204L,203L,202L,201L,200L,199L,198L,197L,196L,195L,194L,193L,192L,191L,190L,189L,188L,187L,186L,185L,184L,183L,182L,181L,180L,179L,178L,177L,176L,175L,174L,173L,172L,171L,168L,167L,166L,165L,164L,163L,162L,161L,160L,159L,158L,157L,155L,154L,153L,152L,151L,150L,149L,148L,147L,146L,145L,144L,143L,142L,141L,140L,139L,135L,134L,132L,133L,131L,130L,129L,128L,127L,126L,125L,124L,123L,122L,121L,120L,119L,118L,117L,116L,114L,113L,112L,111L,110L,109L,108L,107L,106L,105L,104L,103L,102L,101L,99L,98L,97L,96L,84L,83L,80L,77L,76L,75L,74L,73L,72L,70L,71L,69L,68L,67L,66L,65L,62L,60L,64L,44L,43L,42L,41L,39L,38L,18L,17L,15L,14L,13L,12L,8L,4L,3L,2L]
thread-contention-monitoring-supported=true
thread-cpu-time-supported=true
current-thread-cpu-time-supported=true
object-monitor-usage-supported=true
synchronizer-usage-supported=true
thread-contention-monitoring-enabled=false
thread-cpu-time-enabled=true
thread-count=222
peak-thread-count=223
total-started-thread-count=551
daemon-thread-count=147
current-thread-cpu-time=62992810
current-thread-user-time=50000000
object-name=java.lang:type=Threading
[standalone@127.0.0.1:8706 /] ls /core-service=platform-mbean/type=memory
heap-memory-usage={"init" => 1073741824L,"used" => 801489408L,"committed" => 2016411648L,"max" => 2016411648L}
non-heap-memory-usage={"init" => 2555904L,"used" => 194310080L,"committed" => 212074496L,"max" => -1L}
object-name=java.lang:type=Memory
object-pending-finalization-count=0
verbose=true
