Questions tagged [benchmarking]

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them. The benchmarking tag should be used for questions about how to perform benchmarking tasks or theory questions, not for lists of benchmarking results or requests for benchmarking data; those questions are off-topic for Stack Overflow.

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them.

For hardware, benchmarking typically involves either performing a simple task many times or a complex task to ascertain the performance characteristics desired (often speed, but power draw, heat, memory usage, and other characteristics may be of interest as well). Common benchmark operations include FLOPS (Floating Point Operations Per Second), write or read time for a large file, write or read time for many small files, rendering large images, and downloading or uploading large files over a network.

For software, benchmarking typically involves running the different software of interest (different versions of a program, different programs that accomplish a similar task, etc.) on an identical system (either one system, or two identical systems) and performing tasks that take sufficient time to notice a difference. This is often performed on very small differences in code, such as to verify which approach is superior to solving a particular problem.

Benchmarking also includes industry standard benchmarks and common benchmarking suites, used to assist users in making purchasing decisions or otherwise comparing their current systems to other available systems. However, it should be used in this context for issues with building or understanding the code and behavior of these benchmarks, not for general recommendations on the benchmarks suites or on the tested products.

Include additional tags to your question to indicate what type of benchmarking the question is about, such as , or what type of programming language if software, etc.

3517 questions
32
votes
3 answers

What is an idiomatic way to have shared utility functions for integration tests and benchmarks?

I have Rust project with both integration tests (in the /tests dir) and benchmarks (in the /benches dir). There are a couple of utility functions that I need in tests and benches, but they aren't related to my crate itself, so I can't just put them…
Constantine
  • 1,802
  • 3
  • 23
  • 37
32
votes
8 answers

Benchmarks for Intel C++ compiler and GCC

I have an AMD Opteron server running CentOS 5. I want to have a compiler for a fairly large C++ Boost based program. Which compiler I should choose?
James Bond
  • 7,533
  • 19
  • 50
  • 64
32
votes
2 answers

benchmarks: does python have a faster way of walking a network folder?

I need to walk through a folder with approximately ten thousand files. My old vbscript is very slow in handling this. Since I've started using Ruby and Python since then, I made a benchmark between the three scripting languages to see which would…
peter
  • 41,770
  • 5
  • 64
  • 108
31
votes
6 answers

algorithmic trading simulator/benchmark data

I am interested about playing with algorithmic trading strategies. Does anyone know if there exists simulator or benchmark data I could possibly play with offline (without actually making any investments)?
Bob
  • 10,741
  • 27
  • 89
  • 143
31
votes
8 answers

Testing IO performance in Linux

How do I test IO performance in Linux?
deeplovepan
  • 769
  • 2
  • 6
  • 9
30
votes
7 answers

What's the best way to determine at runtime if a browser is too slow to gracefully handle complex JavaScript/CSS?

I'm toying with the idea of progressively enabling/disabling JavaScript (and CSS) effects on a page - depending on how fast/slow the browser seems to be. I'm specifically thinking about low-powered mobile devices and old desktop computers -- not…
Már Örlygsson
  • 14,176
  • 3
  • 42
  • 53
30
votes
1 answer

What does OpenJDK JMH "score error" exactly mean?

I am using http://openjdk.java.net/projects/code-tools/jmh/ for benchmarking and i get a result like: Benchmark Mode Samples Score Score error Units o.a.f.c.j.b.TestClass.test1 avgt 5 2372870,600 …
salyh
  • 2,095
  • 1
  • 17
  • 31
30
votes
4 answers

Get local time in nanoseconds

Possible Duplicate: C++ Timer function to provide time in nano seconds I need to measure the duration of a function execution in nanoseconds resolution. Is it possible? Our ordinary computer hardware and software are able to give such a precision…
Narek
  • 38,779
  • 79
  • 233
  • 389
29
votes
2 answers

F# seems slower than other languages... what can I do to speed it up?

I like F# ; I really, really do. Having been bitten by the "functional programming"-bug, I force myself to use it when I have the opportunity to. In fact, I recently used it (during a one week vacation) to code a nice AI algorithm. However, my…
ttsiodras
  • 10,602
  • 6
  • 55
  • 71
29
votes
5 answers

How to benchmark memory usage of a function?

I notice that Rust's test has a benchmark mode that will measure execution time in ns/iter, but I could not find a way to measure memory usage. How would I implement such a benchmark? Let us assume for the moment that I only care about heap memory…
llogiq
  • 13,815
  • 8
  • 40
  • 72
29
votes
2 answers

Python Numpy Data Types Performance

So I did some testing and got odd results. Code: import numpy as np import timeit setup = """ import numpy as np A = np.ones((1000,1000,3), dtype=datatype) """ datatypes = "np.uint8", "np.uint16", "np.uint32", "np.uint64", "np.float16",…
Gonzo
  • 2,023
  • 3
  • 21
  • 30
28
votes
5 answers

How fast are lambda functions in GCC

Having toyed around a bit with C++0x Lambda Expression in G++, I was wondering as to how well the performance will be in general/specific situations compared to alternative ways without using lambda functions. Does anybody know a more or less…
dcn
  • 4,389
  • 2
  • 30
  • 39
28
votes
2 answers

Does using an intermediate variable instead of array.length make your for-loop faster?

The "Performance Tips" section in the Android documentation has a pretty bold claim: one() is faster. It pulls everything out into local variables, avoiding the lookups. Only the array length offers a performance benefit. where it refers to this…
Jeroen Vannevel
  • 43,651
  • 22
  • 107
  • 170
28
votes
6 answers

Go language benchmarks?

I see the claims that Go is supposed to be almost comparable in speed to C, but are there any benchmarks available yet?
Richard Simões
  • 12,401
  • 6
  • 41
  • 50
27
votes
2 answers

Best method to copy texture to texture

What is the best method to copy pixels from texture to texture? I've found some ways to accomplish this. For instance, there's a method glCopyImageSubData() but my target version is OpenGL 2.1, so I cannot use it. Also, because the performance is…
slyx
  • 2,063
  • 1
  • 19
  • 28