Questions tagged [benchmarking]

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them. The benchmarking tag should be used for questions about how to perform benchmarking tasks or theory questions, not for lists of benchmarking results or requests for benchmarking data; those questions are off-topic for Stack Overflow.

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them.

For hardware, benchmarking typically involves either performing a simple task many times or a complex task to ascertain the performance characteristics desired (often speed, but power draw, heat, memory usage, and other characteristics may be of interest as well). Common benchmark operations include FLOPS (Floating Point Operations Per Second), write or read time for a large file, write or read time for many small files, rendering large images, and downloading or uploading large files over a network.

For software, benchmarking typically involves running the different software of interest (different versions of a program, different programs that accomplish a similar task, etc.) on an identical system (either one system, or two identical systems) and performing tasks that take sufficient time to notice a difference. This is often performed on very small differences in code, such as to verify which approach is superior to solving a particular problem.

Benchmarking also includes industry standard benchmarks and common benchmarking suites, used to assist users in making purchasing decisions or otherwise comparing their current systems to other available systems. However, it should be used in this context for issues with building or understanding the code and behavior of these benchmarks, not for general recommendations on the benchmarks suites or on the tested products.

Include additional tags to your question to indicate what type of benchmarking the question is about, such as , or what type of programming language if software, etc.

3517 questions
39
votes
4 answers

Difference between Benchmarking and Profiling

I see the terms software benchmarking and profiling used sometimes interchangeably but as far as my understanding goes there's a subtile difference. Both are connected by time. But whereas benchmarking is mainly about determining a certain speed…
Jim McAdams
  • 1,034
  • 1
  • 14
  • 30
39
votes
1 answer

Measurement of TLB effects on a Cortex-A9

After reading the following paper https://people.freebsd.org/~lstewart/articles/cpumemory.pdf ("What every programmer should know about memory") I wanted to try one of the author's test, that is, measuring the effects of TLB on the final execution…
D4l3k
  • 575
  • 3
  • 13
39
votes
2 answers

Guava ImmutableMap has noticeably slower access than HashMap

While working on a memory benchmark of some high-throughput data structures, I realized I could use an ImmutableMap with only a little refactoring. Thinking this would be an improvement, I threw it into the mix and was surprised to discover that…
dimo414
  • 47,227
  • 18
  • 148
  • 244
39
votes
4 answers

POST request with wrk?

I started to try wrk. It's very simple to use and very hard on the server, but I don't know how to perform other kind of request such as POST. In fact, I don't even know if this tool allows it. The documentation is very minimal. Thanks
jackdbernier
  • 1,542
  • 1
  • 17
  • 32
39
votes
3 answers

High-Performance Timer vs StopWatch

Does anyone know if the HiPerfTimer or the StopWatch class is better for benchmarking, and why?
Kane
  • 16,471
  • 11
  • 61
  • 86
38
votes
6 answers

HTTPS vs HTTP speed comparison

Update 2013-04-25: This is a popular question that is getting more attention than it probably should. In order to stop the spread of misinformation please read the following paragraphs and the accompanying article first: Speed should not be a factor…
David Murdoch
  • 87,823
  • 39
  • 148
  • 191
36
votes
4 answers

How to load data quickly into R?

I have some R scripts, where I have to load several dataframe in R as quickly as possible. This is quite important as reading the data is the slowest part of the procedure. E.g.: plotting from different dataframes. I get the data in sav (SPSS)…
daroczig
  • 28,004
  • 7
  • 90
  • 124
36
votes
6 answers

Comparing times with sub-second accuracy

How can I get the number of milliseconds since epoch? Note that I want the actual milliseconds, not seconds multiplied by 1000. I am comparing times for stuff that takes less than a second and need millisecond accuracy. (I have looked at lots of…
user984003
  • 28,050
  • 64
  • 189
  • 285
36
votes
7 answers

Benchmarking Performance on Java VM vs .NET CLR

Have you ever had to justify the choice over using .NET instead of Java based on performance? For a typical high volume transaction processing system that can perform the following operations, Concurrent Database transactions Mathematical…
bunn_online
  • 742
  • 1
  • 6
  • 11
35
votes
1 answer

Comparing BSXFUN and REPMAT

Few questions were asked before on comparisons between bsxfun and repmat for performance. One of them was: Matlab - bsxfun no longer faster than repmat?. This one tried to investigate performance comparisons between repmat and bsxfun, specific to…
Divakar
  • 218,885
  • 19
  • 262
  • 358
35
votes
4 answers

C#: Is this benchmarking class accurate?

I created a simple class to benchmark some methods of mine. But is it accurate? I am kind of new to benchmarking, timing, et cetera, so thought I could ask for some feedback here. Also, if it is good, maybe somebody else can make use of it as well…
Svish
  • 152,914
  • 173
  • 462
  • 620
33
votes
2 answers

Is it possible to force an existing Java application to use no more than x cores?

We are benchmarking existing Java programs. They are threaded applications designed to benefit from multi-core CPUs. We would like to measure the effect of the number of cores on the running speed, but we are unwilling (and unable) to change the…
Daniel Lemire
  • 3,470
  • 2
  • 25
  • 23
33
votes
3 answers

Why is reading one byte 20x slower than reading 2, 3, 4, ... bytes from a file?

I have been trying to understand the tradeoff between read and seek. For small "jumps" reading unneeded data is faster than skipping it with seek. While timing different read/seek chunk sizes to find the tipping point, I came across a odd…
ARF
  • 7,420
  • 8
  • 45
  • 72
33
votes
3 answers

What is the purpose of JMH @Fork?

If IIUC each fork creates a separate virtual machine for the reason that each virtual machine instance might run with slight differences in JIT instructions? I'm also curious about what the time attribute does in the below…
Ole
  • 41,793
  • 59
  • 191
  • 359
33
votes
2 answers

Ruby Benchmark module: meanings of "user", "system", and "real"?

Experimenting with Ruby's Benchmark module... >> Benchmark.bm(7) { |b| b.report('Report:') { s = '' ; 10000.times { s += 'a' } } } user system total real Report: 0.150000 0.010000 0.160000 ( 0.156361) What are…
Ethan
  • 57,819
  • 63
  • 187
  • 237