Difference in performance: local machine VS amazon medium instance

Question

I see a drastic difference in performance matrix when i run it with apache benchmark (ab) in my local machine VS production hosted in amazon medium instance. Same concurrent requests (5) and same total number of requests (111) has been run against both. Amazon has better memory than my local machine. But there are 2 CPUs in my local machine vs 1 CPU in m1.medium. My internet speed is very low at the moment, I am getting Transfer rate as 25.29KBps. How can I improve the performance ?

Do not know how to interpret Connect, Processing, Waiting and total in ab output.

Here is Localhost:

Server Hostname:        localhost
Server Port:            9999

Document Path:          /
Document Length:        7631 bytes

Concurrency Level:      5
Time taken for tests:   1.424 seconds
Complete requests:      111
Failed requests:        102
   (Connect: 0, Receive: 0, Length: 102, Exceptions: 0)
Write errors:           0
Total transferred:      860808 bytes
HTML transferred:       847155 bytes
Requests per second:    77.95 [#/sec] (mean)
Time per request:       64.148 [ms] (mean)
Time per request:       12.830 [ms] (mean, across all concurrent requests)
Transfer rate:          590.30 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.5      0       1
Processing:    14   63  99.9     43     562
Waiting:       14   60  96.7     39     560
Total:         14   63  99.9     43     563


And this is production:
Document Path:          /
Document Length:        7783 bytes

Concurrency Level:      5
Time taken for tests:   33.883 seconds
Complete requests:      111
Failed requests:        0
Write errors:           0
Total transferred:      877566 bytes
HTML transferred:       863913 bytes
Requests per second:    3.28 [#/sec] (mean)
Time per request:       1526.258 [ms] (mean)
Time per request:       305.252 [ms] (mean, across all concurrent requests)
Transfer rate:          25.29 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      290  297  14.0    293     413
Processing:   897 1178  63.4   1176    1391
Waiting:      296  606 135.6    588    1171
Total:       1191 1475  66.0   1471    1684

Answer 1

First of all, having 2 CPU's versus 1 CPU is definitely a big difference. Depending on a lot of other factors, that alone could explain some performance difference. However, the big issue here is the connection.

As I understand it, you're wanting to know why the Amazon instance is so much slower than your local machine? Even though the hardware specs suggest they should be relatively comparable? I'm assuming that you have the same setup running locally, and on an Amazon EC2 instance. When you run your benchmark test, you're running it against both from a local machine. Additionally, we can see that your Internet connection speed is rather slow.

What happens is that the time to run the test across the network is making the whole test take a lot longer than running it local. The full answer is a lot longer than I can put here (it involves the way web requests and HTTP works, along with how TCP works, its 3 way handshake for establishing a connection, and how latency and throughput limitations affect network performance). The short answer is that you aren't testing the hardware against each other when you run that test, you're testing the network connectivity. And LAN will (generally) always bean WAN.

This also illustrates a serious pitfall that catches a lot of developers (web and not; anyone with apps that communicate across a network). People will often test their website or app or whatever while it runs on a machine right next to them, and think, "Wow! That's amazingly fast. I'm awesome!". What they forget to do is test it under real world conditions. Just because you have a 23" widescreen monitor doesn't mean that everyone does. Just because you run your web browser full screen doesn't mean everyone does. Sure, loading data is fast across the LAN, but what about for someone halfway around the world? These are the kinds of things that need to be considered when developing a webpage or network application.