Questions tagged [benchmarking]

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them. The benchmarking tag should be used for questions about how to perform benchmarking tasks or theory questions, not for lists of benchmarking results or requests for benchmarking data; those questions are off-topic for Stack Overflow.

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them.

For hardware, benchmarking typically involves either performing a simple task many times or a complex task to ascertain the performance characteristics desired (often speed, but power draw, heat, memory usage, and other characteristics may be of interest as well). Common benchmark operations include FLOPS (Floating Point Operations Per Second), write or read time for a large file, write or read time for many small files, rendering large images, and downloading or uploading large files over a network.

For software, benchmarking typically involves running the different software of interest (different versions of a program, different programs that accomplish a similar task, etc.) on an identical system (either one system, or two identical systems) and performing tasks that take sufficient time to notice a difference. This is often performed on very small differences in code, such as to verify which approach is superior to solving a particular problem.

Benchmarking also includes industry standard benchmarks and common benchmarking suites, used to assist users in making purchasing decisions or otherwise comparing their current systems to other available systems. However, it should be used in this context for issues with building or understanding the code and behavior of these benchmarks, not for general recommendations on the benchmarks suites or on the tested products.

Include additional tags to your question to indicate what type of benchmarking the question is about, such as , or what type of programming language if software, etc.

3517 questions
49
votes
6 answers

Java benchmarking - why is the second loop faster?

I'm curious about this. I wanted to check which function was faster, so I create a little code and I executed a lot of times. public static void main(String[] args) { long ts; String c = "sgfrt34tdfg34"; ts =…
Guille
  • 2,248
  • 3
  • 24
  • 42
48
votes
3 answers

Which gets the measurements right, JMeter or Apache ab?

I started writing some basic tests in JMeter and was surprised that the measurements are so different from those from Apache ab. I have a gigabit LAN connecting an Intel i7 server running Nginx and an i5 test machine running JMeter or ab.…
Rick-777
  • 9,714
  • 5
  • 34
  • 50
45
votes
2 answers

How does jsPerf determine which of the code snippets is fastest?

Today I visited jsPerf and now I am wondering… What is "ops/sec"? How many iterations does it do? On what basis does it calculate which is faster? What is the formula behind these calculations? Example: http://jsperf.com/concatenation-vs-join Can…
Mohit Kumar
  • 1,885
  • 5
  • 21
  • 24
45
votes
3 answers

How fast is Berkeley DB SQL compared to SQLite?

Oracle recently released a Berkeley DB back-end to SQLite. I happen to have a hundreds-of-megabytes SQLite database that could very well benefit from "improved performance, concurrency, scalability, and reliability", but Oracle's site appears to…
dan04
  • 87,747
  • 23
  • 163
  • 198
45
votes
1 answer

Measure (max) memory usage with IPython—like timeit but memit

I have a simple task: in addition to measuring the time it takes to execute a chunk of code in Python, I need to measure the amount of memory a given chunk of code needs. IPython has a nice utility called timeit which works like this: In [10]:…
Erik Kaplun
  • 37,128
  • 15
  • 99
  • 111
45
votes
2 answers

Why is Symfony2 performing so bad in benchmarks and does it matter?

My colleagues and I are in the process of choosing a web framework to develop a high traffic web site. We are really good with node.js + express and php + symfony2. Both are great frameworks but we are a bit concerned about Symfony2 because it seems…
Jean-Philippe Leclerc
  • 6,713
  • 5
  • 43
  • 66
45
votes
3 answers

Quantifiable metrics (benchmarks) on the usage of header-only c++ libraries

I've tried to find an answer to this using SO. There are a number of questions that list the various pros and cons of building a header-only library in c++, but I haven't been able to find one that does so in quantifiable terms. So, in quantifiable…
Homer6
  • 15,034
  • 11
  • 61
  • 81
44
votes
5 answers

How am I supposed to interpret the results from Apache's ab benchmarking tool?

Alright, I've searched everywhere and I can't seem to find a detailed resource online for how to interpret the results from Apache's ab server benchmarking tool. I've run several tests with what I thought were drastically different parameters, but…
Nick
  • 1,311
  • 2
  • 10
  • 27
44
votes
1 answer

Does Rust's array bounds checking affect performance?

I'm coming from C and I wonder whether Rust's bounds checking affects performance. It probably needs some additional assembly instructions for every access, which could hurt when processing lots of data. On the other hand, the costly thing in…
Jounathaen
  • 803
  • 1
  • 9
  • 23
44
votes
6 answers

Why is BufferedReader read() much slower than readLine()?

I need to read a file one character at a time and I'm using the read() method from BufferedReader. * I found that read() is about 10x slower than readLine(). Is this expected? Or am I doing something wrong? Here's a benchmark with Java 7. The input…
dariober
  • 8,240
  • 3
  • 30
  • 47
42
votes
4 answers

Why is 2 * x * x faster than 2 * ( x * x ) in Python 3.x, for integers?

The following Python 3.x integer multiplication takes on average between 1.66s and 1.77s: import time start_time = time.time() num = 0 for x in range(0, 10000000): # num += 2 * (x * x) num += 2 * x * x print("--- %s seconds ---" %…
42
votes
3 answers

How to benchmark functions in Clojure?

I know I can get the time take to evaluate a function can be printed out on the screen/stdout using the time function/macro. The time macro returns the value of the evaluated function, which makes it great to use it inline. However I want to…
Peter Tillemans
  • 34,983
  • 11
  • 83
  • 114
40
votes
5 answers

How do you time a function in Go and return its runtime in milliseconds?

How do you time a function in Go and return its runtime in milliseconds?
The Puma
  • 1,352
  • 2
  • 14
  • 27
40
votes
1 answer

What does "an intermediate result is being cached" mean?

I have a set of n vectors stored in the 3 x n matrix z. I find the outer product using np.einsum. When I timed it using: %timeit v=np.einsum('i...,j...->ij...',z,z) I got the result: The slowest run took 7.23 times longer than the fastest. This…
user3799584
  • 917
  • 1
  • 9
  • 18
40
votes
3 answers

How to test only one benchmark function?

In my Go package there are several benchmark files like map1_benchmark_test.go and map2_benchmark_test.go. In every *_benchmark_test.go file, there is more than one benchmark function like func BenchmarkMapTravel(b *testing.B) and func…
hardPass
  • 19,033
  • 19
  • 40
  • 42