I've been rethinking the compiler shootout and I began to wonder: is Amazon EC suitable for benchmarking? The environment is totally virtual, so are there any guarantees that benchmarking results will be repeatable? Or should I assume that the virtual hosting environment time slicing interferes with timing approaches?
Asked
Active
Viewed 790 times
2 Answers
1
No.
Not that this is EC2's fault, but when benchmarking, when comparing A and B, one needs to keep in mind the phrase "all things being equal". (There is a formal name for this, but I forget.) With the EC2 virtualized environment, where you are not in control of anything but your instance, this is just not the case.
Unless you are benchmarking cloud environments ;)
(Edited to be specific to EC2, thanks to knowledgeable comments)
Stu Thompson
- 3,349
- 7
- 31
- 47
-
s/visualized/virtualized – jldugger May 07 '09 at 18:50
-
I don't think that cloud computing necessarily means its impossible to offer strict guarentees about availability. Probably the main challenge comes from NOT giving a CPU intensive VM the slack time. – jldugger May 07 '09 at 18:53
-
http://en.wikipedia.org/wiki/Ceteris_paribus – jldugger May 07 '09 at 18:54
-
1Fair enough, folks. I will qualify my answer as EC2 specific. I've done some benchmarking on EC2, mostly bandwidth and response time oriented, and was surprised at how variable things are. All things are not equal. – Stu Thompson May 07 '09 at 19:01
-
1@jldugger: exit 0 – Stu Thompson May 07 '09 at 19:07
0
In benchmarking, you want to control all of the variables you can to ensure proper testing. In a cloud computing environment, I would not trust that a given instance will be exactly the same from one moment to the next enough to perform proper benchmarking reliably.
Justin Scott
- 8,798
- 1
- 28
- 39