We are load testing our application through jmeter.
The architecture is : web server (Apache), app server (jboss), database(MySQL).
Dynatrace agent is configured in app server.
For any particular request, the response time shown in app server is less than 1 sec, but in jmeter report it shows more than 5 sec.
What could be possible reasons for above behaviour.
What should be next steps which I should perform to find bottleneck.
Check this glossary, the first three params described (Elapsed Time, Latency, Connect Time).
That is what JMeter is measuring on it's side - which is a client side, obviously. While Dynatrace measures server-side processing only.
So both numbers are legit and gives you an important clue - you just have to read it properly.
Armed with that numbers, you'd probably like to deep into what happened in between, where this difference is coming from.
In a nutshell, in between lays: network; standard components of the server (OS & JBoss), including ones that serves networking stuff; same for client; and JMeter itself.
To get the first clue on that path, I suggest you to set up the sniffing proxy (e.g. Fiddler) on your JMeter host, manage all traffic to go through it, and see what it shows in its detailed time statistics for the networking operations.
This way, you'd have your JMeter excluded as a factor, so you can see how much it brings there.
Of course it's different and it's normal !
Processing Time for (request + response) between JBoss and Database take 1s.
Processing Time for (request + response) between JMeter and Database take 5s.
Response Time displayed by Dynatrace is the time calculated between Jboss an Database (time spent between the first request and the last response).
Response Time displayed by JMeter is the time calculated between JMeter and the Database : The time integrate Response Time between JMeter and your Jboss + RT between Jboss and Database. It must be greater than Response Time from Dynatrace as in your case (5 seconds against 1 second).
What does it mean ?
It means that you lose time between JMeter and Jboss (5-1 = 4 seconds) , 4 more time than time processing between WAS and Database => tuning of your Web Server and your WAS could reduce the time spent in a first time...
