Questions tagged [benchmarking]

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them. The benchmarking tag should be used for questions about how to perform benchmarking tasks or theory questions, not for lists of benchmarking results or requests for benchmarking data; those questions are off-topic for Stack Overflow.

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them.

For hardware, benchmarking typically involves either performing a simple task many times or a complex task to ascertain the performance characteristics desired (often speed, but power draw, heat, memory usage, and other characteristics may be of interest as well). Common benchmark operations include FLOPS (Floating Point Operations Per Second), write or read time for a large file, write or read time for many small files, rendering large images, and downloading or uploading large files over a network.

For software, benchmarking typically involves running the different software of interest (different versions of a program, different programs that accomplish a similar task, etc.) on an identical system (either one system, or two identical systems) and performing tasks that take sufficient time to notice a difference. This is often performed on very small differences in code, such as to verify which approach is superior to solving a particular problem.

Benchmarking also includes industry standard benchmarks and common benchmarking suites, used to assist users in making purchasing decisions or otherwise comparing their current systems to other available systems. However, it should be used in this context for issues with building or understanding the code and behavior of these benchmarks, not for general recommendations on the benchmarks suites or on the tested products.

Include additional tags to your question to indicate what type of benchmarking the question is about, such as , or what type of programming language if software, etc.

3517 questions
2
votes
2 answers

Benchmarking and restricting execution to a specific CPU

How can I constrain all my benchmarks to run on a single CPU (e.g. C0)? I am running benchmarks and want to expose my tests to an environment similar to the target. I would also like an advice as to how I could ensure that there are minimal other…
Walker
  • 323
  • 1
  • 12
2
votes
3 answers

Testing speed of code execution with Java/Eclipse

I want to test which of the two methods executes faster, however depending on which one I run first, that method seems to always run slower. I suspect there is some caching mechanism Eclipse is using that speeds up the execution of the second…
budi
  • 6,351
  • 10
  • 55
  • 80
2
votes
0 answers

Is there an option for mysql as dialect while generating queries for TPC-DS?

I have loaded the data and everything is set. But the dialects available are db2, oracle, sqlserver, ansi and netezza. None of these are fully compatible with mysql. Is there a way to generate the queries for mysql?
Kumar Deepak
  • 473
  • 4
  • 18
2
votes
1 answer

Cassandra Reading Benchmark with Spark

I'm doing a benchmark on Cassandra's Reading performance. In the test-setup step I created a cluster with 1 / 2 / 4 ec2-instances and data nodes. I wrote 1 table with 100 million of entries (~3 GB csv-file). Then I launch a Spark application which…
2
votes
1 answer

What is the maxium replication rate of Couchbase XDCR

We are currently using Couchbase for data caching and there is talk of doing cross-data center replication with it. However, we will need up to 1000 documents replicated to multiple locations every second. Documents will be between 2 and 64K…
2
votes
2 answers

How to measure if a program was run in parallel over multiple cores in Linux?

I want to know if my program was run in parallel over multiple cores. I can get the perf tool to report how many cores were used in the computation, but not if they were used at the same time (in parallel). How can this be done?
ivarec
  • 2,542
  • 2
  • 34
  • 57
2
votes
1 answer

Is there a cost to jQuerying an existing jQuery object?

I'm wondering whether lazily doing $( something ) without first checking whether something is already a jQuery instance has any performance downsides? The use case I'm thinking of is a function like the following: function foo( element ){ var…
Tom Auger
  • 19,421
  • 22
  • 81
  • 104
2
votes
0 answers

Haskell Criterion Benchmark on lazy encode

We're trying to benchmark binary encoding with Criterion. Since the data types are strict, we are perfectly able to benchmark the process of packing a Request. However, one step further by trying to benchmark the process of encode (Request to…
Marc Juchli
  • 2,240
  • 24
  • 20
2
votes
1 answer

Performance gain from double counter in a js "for" loop?

While researching array assignment performance, I stumbled across this jsperf which purportedly shows a very dramatic speed increase for for loops of the form: var i, j = 0; for (i = 0; i < n; i++) { myArray[j++] = i; } ... in current browsers.…
Ben
  • 11,082
  • 8
  • 33
  • 47
2
votes
2 answers

pre-sorting improves itertools.combinations performances?

I'm currently working on large data, and an important step of the work is to get all sorted combinations of thousand of strings. In python 3.4, i try three ways for perform this operation. First, faking the data as a list of string, with or without…
aluriak
  • 5,559
  • 2
  • 26
  • 39
2
votes
1 answer

Error benchmarking multithreaded algorithm using parameterized 'jmh' measurement

For benchmarking a multithreaded algorithm, I set up a parameterized jmh measurement. After quite some time, the measurement crashes with the error java.lang.OutOfMemoryError: unable to create new native thread The algorithm creates a very limited…
bisgardo
  • 4,130
  • 4
  • 29
  • 38
2
votes
1 answer

Benchmark for loop and range operator

I read that range-based loops have better performance on some programming language. Is it the case in Swift. For instance in Playgroud: func timeDebug(desc: String, function: ()->() ) { let start : UInt64 = mach_absolute_time() function() …
grape1
  • 759
  • 2
  • 8
  • 19
2
votes
1 answer

Measuring execution time of other processes with C#, odd results

I'm trying to build a small benchmarking application, which allows the user to measure the execution time and memory usage of a program. This is the code to measure the execution time: private static Stopwatch _stopwatch; static void Main(string[]…
Aifu
  • 21
  • 2
2
votes
1 answer

How do i write a benchmark script in Go to measure ops/sec

I am practicing my Golang by writing a simple Redis clone. How do i write a benchmark script that would establish X connections per second at C concurrency level to deal with my server's protocol and measure how many ops/sec? I can simply write a…
user3470707
  • 183
  • 1
  • 7
2
votes
1 answer

OptaPlanner benchmarking without XML inputSolutionFile

I have developed a working solver, which generates the unsolved solution directly from database(without and XML file). Now I am starting to develop it's benchmarker. Since all the example benchmarkers utilizes inputSolutionFile(xml files), I am…
Pejal Hebat
  • 116
  • 6