Questions tagged [benchmarking]

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them. The benchmarking tag should be used for questions about how to perform benchmarking tasks or theory questions, not for lists of benchmarking results or requests for benchmarking data; those questions are off-topic for Stack Overflow.

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them.

For hardware, benchmarking typically involves either performing a simple task many times or a complex task to ascertain the performance characteristics desired (often speed, but power draw, heat, memory usage, and other characteristics may be of interest as well). Common benchmark operations include FLOPS (Floating Point Operations Per Second), write or read time for a large file, write or read time for many small files, rendering large images, and downloading or uploading large files over a network.

For software, benchmarking typically involves running the different software of interest (different versions of a program, different programs that accomplish a similar task, etc.) on an identical system (either one system, or two identical systems) and performing tasks that take sufficient time to notice a difference. This is often performed on very small differences in code, such as to verify which approach is superior to solving a particular problem.

Benchmarking also includes industry standard benchmarks and common benchmarking suites, used to assist users in making purchasing decisions or otherwise comparing their current systems to other available systems. However, it should be used in this context for issues with building or understanding the code and behavior of these benchmarks, not for general recommendations on the benchmarks suites or on the tested products.

Include additional tags to your question to indicate what type of benchmarking the question is about, such as , or what type of programming language if software, etc.

3517 questions
59
votes
4 answers

clearing a small integer array: memset vs. for loop

There are two ways to zero out an integer/float array: memset(array, 0, sizeof(int)*arraysize); or: for (int i=0; i
Claudiu
  • 224,032
  • 165
  • 485
  • 680
57
votes
4 answers

What's really more performant? Haskell or OCaml

I spent the last 18 months getting the grip of functional programming, starting with learning OCaml and for some weeks now Haskell. Now I want to take the next step and implement some actual application: A simple realtime terrain editor. I've…
datenwolf
  • 159,371
  • 13
  • 185
  • 298
57
votes
1 answer

What does allocs/op and B/op mean in go benchmark?

When I run my benchmarks with go test -v -bench=. -benchmem, I see the following results. f1 10000 120860 ns/op 2433 B/op 28 allocs/op f2 10000 120288 ns/op 2288 B/op 26 allocs/op Based on my…
Salvador Dali
  • 214,103
  • 147
  • 703
  • 753
56
votes
1 answer

Mystifying microbenchmark result for stream API on Java 12 vs. Java 8 with -gc true

As part of my investigation on the difference between using a complex filter or multiple filters in streams, I notice that performance on Java 12 is way slower than on Java 8. Is any explanation for those weird results? Did I miss something here?…
Serge
  • 2,574
  • 18
  • 26
55
votes
7 answers

SQLite Performance Benchmark -- why is :memory: so slow...only 1.5X as fast as disk?

Why is :memory: in sqlite so slow? I've been trying to see if there are any performance improvements gained by using in-memory sqlite vs. disk based sqlite. Basically I'd like to trade startup time and memory to get extremely rapid queries which do…
ramanujan
  • 5,581
  • 5
  • 30
  • 31
53
votes
9 answers

Is if(A | B) always faster than if(A || B)?

I am reading this book by Fedor Pikus and he has some very very interesting examples which for me were a surprise. Particularly this benchmark caught me, where the only difference is that in one of them we use || in if and in another we use |. void…
Eduard Rostomyan
  • 7,050
  • 2
  • 37
  • 76
53
votes
5 answers

Java vs C#: Are there any studies that compare their execution speed?

Taking out all of the obvious caveats related to benchmarks and benchmark comparison, is there any study (an array of well documented and unbiased tests) that compares the average execution speed of the two mentioned languages? Thanks
pistacchio
  • 56,889
  • 107
  • 278
  • 420
52
votes
18 answers

How to measure code performance in .NET?

I'm doing some real quick and dirty benchmarking on a single line of C# code using DateTime: long lStart = DateTime.Now.Ticks; // do something long lFinish = DateTime.Now.Ticks; The problem is in the results: Start Time [633679466564559902] Finish…
Scott Marlowe
  • 7,915
  • 10
  • 45
  • 51
52
votes
9 answers

What is FLOP/s and is it a good measure of performance?

I've been asked to measure the performance of a fortran program that solves differential equations on a multi-CPU system. My employer insists that I measure FLOP/s (Floating operations per second) and compare the results with benchmarks (LINPACK)…
caglarozdag
  • 699
  • 1
  • 8
  • 13
52
votes
2 answers

Is the Julia language really as fast as it claims?

Following this post I decided to benchmark Julia against GNU Octave and the results were inconsistent with the speed-ups illustrated in julialang.org. I compiled both Julia and GNU Octave with CXXFLAGS='-std=c++11 -O3', the results I got: GNU…
juliohm
  • 3,691
  • 2
  • 18
  • 22
51
votes
4 answers

Is the UNIX `time` command accurate enough for benchmarks?

Let's say I wanted to benchmark two programs: foo.py and bar.py. Are a couple thousand runs and the respective averages of time python foo.py and time python bar.py adequate enough for profiling and comparing their speed? Edit: Additionally, if the…
chrisdotcode
  • 1,569
  • 2
  • 17
  • 22
51
votes
6 answers

Why does C# execute Math.Sqrt() more slowly than VB.NET?

Background While running benchmark tests this morning, my colleagues and I discovered some strange things concerning performance of C# code vs. VB.NET code. We started out comparing C# vs. Delphi Prism calculating prime numbers, and found that Prism…
Matt Winckler
  • 2,223
  • 2
  • 23
  • 27
51
votes
4 answers

Getting an accurate execution time in C++ (micro seconds)

I want to get an accurate execution time in micro seconds of my program implemented with C++. I have tried to get the execution time with clock_t but it's not accurate. (Note that micro-benchmarking is hard. An accurate timer is only a small part…
user3323616
  • 513
  • 1
  • 4
  • 4
50
votes
3 answers

Definition of Connect, Processing, Waiting in apache bench

When I run apache bench I get results like: Command: abs.exe -v 3 -n 10 -c 1 https://mysite Connection Times (ms) min mean[+/-sd] median max Connect: 203 213 8.1 219 219 Processing: 78 177 88.1 172 …
rpatel
  • 603
  • 1
  • 5
  • 8
50
votes
7 answers

Ackermann very inefficient with Haskell/GHC

I try computing Ackermann(4,1) and there's a big difference in performance between different languages/compilers. Below are results on my Core i7 3820QM, 16G, Ubuntu 12.10 64bit, C: 1.6s, gcc -O3 (with gcc 4.7.2) int ack(int m, int n) { if (m ==…
Phil
  • 5,595
  • 5
  • 35
  • 55