Questions tagged [benchmarking]

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them. The benchmarking tag should be used for questions about how to perform benchmarking tasks or theory questions, not for lists of benchmarking results or requests for benchmarking data; those questions are off-topic for Stack Overflow.

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them.

For hardware, benchmarking typically involves either performing a simple task many times or a complex task to ascertain the performance characteristics desired (often speed, but power draw, heat, memory usage, and other characteristics may be of interest as well). Common benchmark operations include FLOPS (Floating Point Operations Per Second), write or read time for a large file, write or read time for many small files, rendering large images, and downloading or uploading large files over a network.

For software, benchmarking typically involves running the different software of interest (different versions of a program, different programs that accomplish a similar task, etc.) on an identical system (either one system, or two identical systems) and performing tasks that take sufficient time to notice a difference. This is often performed on very small differences in code, such as to verify which approach is superior to solving a particular problem.

Benchmarking also includes industry standard benchmarks and common benchmarking suites, used to assist users in making purchasing decisions or otherwise comparing their current systems to other available systems. However, it should be used in this context for issues with building or understanding the code and behavior of these benchmarks, not for general recommendations on the benchmarks suites or on the tested products.

Include additional tags to your question to indicate what type of benchmarking the question is about, such as , or what type of programming language if software, etc.

3517 questions
89
votes
6 answers

What is microbenchmarking?

I've heard this term used, but I'm not entirely sure what it means, so: What DOES it mean and what DOESN'T it mean? What are some examples of what IS and ISN'T microbenchmarking? What are the dangers of microbenchmarking and how do you avoid…
polygenelubricants
  • 376,812
  • 128
  • 561
  • 623
87
votes
3 answers

Performance Benchmarking of Contains, Exists and Any

I have been searching for a performance benchmarking between Contains, Exists and Any methods available in the List. I wanted to find this out just out of curiosity as I was always confused among these. Many questions on SO described definitions…
harshit
  • 3,788
  • 3
  • 31
  • 54
86
votes
4 answers

What is the best way to measure execution time of a function?

Obviously I can do and DateTime.Now.After - DateTime.Now.Before but there must be something more sophisticated. Any tips appreciated.
Ian G
  • 29,468
  • 21
  • 78
  • 92
85
votes
5 answers

python: deque vs list performance comparison

In python docs I can see that deque is a special collection highly optimized for poping/adding items from left or right sides. E.g. documentation says: Deques are a generalization of stacks and queues (the name is pronounced “deck” and is short…
Oleg Tarasenko
  • 9,324
  • 18
  • 73
  • 102
84
votes
16 answers

Does Google Analytics have performance overhead?

To what extent does Google Analytics impact performance? I'm looking for the following: Benchmarks (including response times/pageload times et al) Links or results to similar benchmarks One (possible) method of testing Google Analytics (GA) on…
M.N
  • 10,899
  • 13
  • 47
  • 49
83
votes
5 answers

How to use python timeit when passing variables to functions?

I'm struggling with this using timeit and was wondering if anyone had any tips Basically I have a function(that I pass a value to) that I want to test the speed of and created this: if __name__=='__main__': from timeit import Timer t =…
Lostsoul
  • 25,013
  • 48
  • 144
  • 239
79
votes
13 answers

Is stopwatch benchmarking acceptable?

Does anyone ever use stopwatch benchmarking, or should a performance tool always be used? Are there any good free tools available for Java? What tools do you use? To clarify my concerns, stopwatch benchmarking is subject to error due to operating…
Bill the Lizard
  • 398,270
  • 210
  • 566
  • 880
75
votes
11 answers

How can I find the missing value more concisely?

The following code checks if x and y are distinct values (the variables x, y, z can only have values a, b, or c) and if so, sets z to the third character: if x == 'a' and y == 'b' or x == 'b' and y == 'a': z = 'c' elif x == 'b' and y == 'c' or x…
Bunny Rabbit
  • 8,213
  • 16
  • 66
  • 106
74
votes
2 answers

Why does this delay-loop start to run faster after several iterations with no sleep?

Consider: #include #include #include using namespace std; const int times = 1000; const int N = 100000; void run() { for (int j = 0; j < N; j++) { } } int main() { clock_t main_start = clock(); for (int i =…
phyxnj
  • 647
  • 5
  • 11
69
votes
13 answers

Large public datasets?

I am looking for some large public datasets, in particular: Large sample web server logs that have been anonymized. Datasets used for database performance benchmarking. Any other links to large public datasets would be appreciated. I already know…
Jason
69
votes
5 answers

How can I benchmark C code easily?

Is there a simple library to benchmark the time it takes to execute a portion of C code? What I want is something like: int main(){ benchmarkBegin(0); //Do work double elapsedMS = benchmarkEnd(0); benchmarkBegin(1) //Do some…
Mike
  • 23,892
  • 18
  • 70
  • 90
64
votes
6 answers

Why is matrix multiplication faster with numpy than with ctypes in Python?

I was trying to figure out the fastest way to do matrix multiplication and tried 3 different ways: Pure python implementation: no surprises here. Numpy implementation using numpy.dot(a, b) Interfacing with C using ctypes module in Python. This is…
Charles Menguy
  • 40,830
  • 17
  • 95
  • 117
60
votes
4 answers

Python Requests vs PyCurl Performance

How does the Requests library compare with the PyCurl performance wise? My understanding is that Requests is a python wrapper for urllib whereas PyCurl is a python wrapper for libcurl which is native, so PyCurl should get better performance, but…
Eugene
  • 10,957
  • 20
  • 69
  • 97
59
votes
6 answers

Why does JavaScript appear to be 4 times faster than C++?

For a long time, I had thought of C++ being faster than JavaScript. However, today I made a benchmark script to compare the speed of floating point calculations in the two languages and the result is amazing! JavaScript appears to be almost 4 times…
streaver91
  • 864
  • 1
  • 8
  • 10
59
votes
5 answers

How do I get monotonic time durations in python?

I want to log how long something takes in real walltime. Currently I'm doing this: startTime = time.time() someSQLOrSomething() print "That took %.3f seconds" % (time.time() - startTime) But that will fail (produce incorrect results) if the time is…
Thomas
  • 4,208
  • 2
  • 29
  • 31