Questions tagged [benchmarking]

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them. The benchmarking tag should be used for questions about how to perform benchmarking tasks or theory questions, not for lists of benchmarking results or requests for benchmarking data; those questions are off-topic for Stack Overflow.

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them.

For hardware, benchmarking typically involves either performing a simple task many times or a complex task to ascertain the performance characteristics desired (often speed, but power draw, heat, memory usage, and other characteristics may be of interest as well). Common benchmark operations include FLOPS (Floating Point Operations Per Second), write or read time for a large file, write or read time for many small files, rendering large images, and downloading or uploading large files over a network.

For software, benchmarking typically involves running the different software of interest (different versions of a program, different programs that accomplish a similar task, etc.) on an identical system (either one system, or two identical systems) and performing tasks that take sufficient time to notice a difference. This is often performed on very small differences in code, such as to verify which approach is superior to solving a particular problem.

Benchmarking also includes industry standard benchmarks and common benchmarking suites, used to assist users in making purchasing decisions or otherwise comparing their current systems to other available systems. However, it should be used in this context for issues with building or understanding the code and behavior of these benchmarks, not for general recommendations on the benchmarks suites or on the tested products.

Include additional tags to your question to indicate what type of benchmarking the question is about, such as , or what type of programming language if software, etc.

3517 questions
27
votes
3 answers

Writing a time function in Haskell

I’m new to Haskell and I’d like to be able to time the runtime of a given function call or snippet of code. In Clojure I can use ‘time’: user=> (time (apply * (range 2 10000))) "Elapsed time: 289.795…
Jacob
  • 273
  • 3
  • 4
26
votes
1 answer

Why is iterating over std::ranges::views::join so slow

This is a follow-up of this SO Answer. Given a flat input range and three size_t dimensions, the code creates a nested random_access_range of random_access_ranges of random_access_ranges, modelling a three-dimensional array. Quickbench Iterating…
joergbrech
  • 2,056
  • 1
  • 5
  • 17
26
votes
3 answers

when should I use a sorteddictionary instead of a dictionary

As I wrote in some of my last posts I am still quite new to the c# world so it comes that I wrote small benchmark to compare Dictionary, Hashtable, SortedList and SortedDictionary against each other. The test runs with 8000 iterations and from 50 to…
sra
  • 23,820
  • 7
  • 55
  • 89
26
votes
7 answers

How to speed up matrix multiplication in C++?

I'm performing matrix multiplication with this simple algorithm. To be more flexible I used objects for the matricies which contain dynamicly created arrays. Comparing this solution to my first one with static arrays it is 4 times slower. What can…
multiholle
  • 3,050
  • 8
  • 41
  • 60
25
votes
1 answer

Python Slice Assignment Memory Usage

I read in a comment here on Stack Overflow that it is more memory efficient to do slice assignment when changing lists. For example, a[:] = [i + 6 for i in a] should be more memory efficient than a = [i + 6 for i in a] because the former replaces…
Mitch Lindgren
  • 2,120
  • 1
  • 18
  • 36
25
votes
3 answers

How can I benchmark the performance of C++ code?

I am starting to study algorithms and data structures seriously, and interested in learning how to compare the performance of the different ways I can implement A&DTs. For simple tests, I can get the time before/after something runs, run that thing…
user8762495
25
votes
5 answers

Looking for benchmarking code snippet (c++)

Some loading routines in my program takes to long to complete. I want a quick small snippet for checking how long a function took to execute. By small I mean "preferably without 3rd party libraries". Maybe something as simple as taking the system…
Mizipzor
  • 51,151
  • 22
  • 97
  • 138
25
votes
3 answers

Why threads are showing better performance than coroutines?

I have written 3 simple programs to test coroutines performance advantage over threads. Each program does a lot of common simple computations. All programs were run separately from each other. Besides execution time I measured CPU usage via Visual…
Praytic
  • 1,771
  • 4
  • 21
  • 41
25
votes
2 answers

UDP send performance in Node.js

I am benchmarking a Java UDP client that continuously sends datagrams with a payload of 100 bytes as fast as it can. It was implemented using java.nio.*. Tests show that it's able to achieve a steady throughput of 220k datagrams per second. I am not…
Lucio Paiva
  • 19,015
  • 11
  • 82
  • 104
25
votes
3 answers

Array vs Slice: accessing speed

This question is about the speed of accessing elements of arrays and slices, not about the efficiency of passing them to functions as arguments. I would expect arrays to be faster than slices in most cases because a slice is a data structure…
icza
  • 389,944
  • 63
  • 907
  • 827
25
votes
2 answers

Is A==0 really better than ~A?

Introduction to problem setup I was doing some benchmarks involving - ~A and A==0for a double array with no NaNs, both of which convert A to a logical array where all zeros are converted to true values and rest are set as false values. For the…
Divakar
  • 218,885
  • 19
  • 262
  • 358
25
votes
4 answers

Java 2D array fill - innocent optimization caused terrible slowdown

I've tried to optimize a filling of square two-dimensional Java array with sums of indices at each element by computing each sum once for two elements, opposite relative to the main diagonal. But instead of speedup or, at least, comparable…
leventov
  • 14,760
  • 11
  • 69
  • 98
24
votes
1 answer

numpy.cos works significantly longer on certain numbers

TLDR: numpy.cos() works 30% longer on a particular numbers (exactly 24000.0 for example). Adding a small delta (+0.01) causes numpy.cos() to work as usual. I have no idea why. I stumbled across a strange problem during my work with numpy. I was…
vurmux
  • 9,420
  • 3
  • 25
  • 45
24
votes
7 answers

How to benchmark php/mysql site

I would like to know how to benchmark a php/mysql site. We have a web app almost completed and ready to go live, we know how many people are going to be using it in a years time but have absolutely no idea how much bandwidth the average user takes,…
user103219
  • 3,209
  • 11
  • 39
  • 50
23
votes
4 answers

get execution time in milliseconds in R

I have read a solution to this using tic(), toc() functions tic <- function(gcFirst = TRUE, type=c("elapsed", "user.self", "sys.self")) { type <- match.arg(type) assign(".type", type, envir=baseenv()) if(gcFirst) gc(FALSE) tic <-…
edgarmtze
  • 24,683
  • 80
  • 235
  • 386