5

I am trying to run a particular test multiple times in ipython using the %timeit magic function. For demonstration purposes, I will just use -n1 instead of -n3 here, and use a simple print(1) function.

The %%timeit and the %timeit help says the following:

Options: -n<N>: execute the given statement <N> times in a loop. If this
value is not given, a fitting value is chosen.

-r<R>: repeat the loop iteration <R> times and take the best result.
Default: 3 (the 3 here is a typo in ipython, for which I have submitted a
PR)

However, if I do the following:

%%timeit -n1
print(1)

or

%timeit -n1 print(1)

it actually prints 1 7 times in a row as follows

In[1]: %timeit -n1 print(1)
1
1
1
1
1
1
1
32.8 µs ± 38.7 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

I was expecting that because of the definition of %%timeit/%timeit it would run the cell or code just once.

I have read this post: https://stackoverflow.com/a/45375047/4752883 which gives some examples of how %%timeit runs and the actual source code of ipython magic function %%timeit here: https://github.com/ipython/ipython/blob/ec3d1a11bf26a0962cb2cf55ba263b12ac023183/IPython/core/magics/execution.py#L944

where they define 2 types of loops: 1) -n<N> and 2) -r<R>.

If I just use -n1, it seems that it also assumes that I have used -r7, i.e. -n1 defaults to -n1 -r7. This means that even though I want it to run for exactly 1 run, it will still run the code_block 7 times as per https://github.com/ipython/ipython/blob/ec3d1a11bf26a0962cb2cf55ba263b12ac023183/IPython/core/magics/execution.py#L1021 unless I also specify -n1 -r1.

Questions:

  1. Why are there 2 different ways to running the code_block using -n<N> and -r<R>?
  2. What is the difference between -n<N> and -r<R> and why is this necessary?
alpha_989
  • 4,882
  • 2
  • 37
  • 48

1 Answers1

7

These parameters are also in the timeit module.

  • -n determines how many times you run the function (or block, or whatever) inside the timing window. So the stopwatch starts, the code is run n times, then the stopwatch ends. You should run it enough times that the results are meaningful (timeit defaults to powers of 10 until 0.2 seconds have elapsed).
  • -r determines how many of these repetitions (where repetition is "start timer, run n times, stop timer") you should do. There is always some error due to your CPU scheduling other processes, etc., so usually you want to run it a few times and the best value of these r times is taken. (timeit defaults to 3, and the comments in the source you link indicate that ipython does the same -- but the actual code may disagree).

In pseudo-python, you can see how n and r affect the timing process:

time_hist = []
for _ in range(r):
    t0 = time.now()              # Start stopwatch (.now() is not a real function)
    for _ in range(n):
        # <your code block>
    t1 = time.now()              # Stop stopwatch

    time_hist.append(t1 - t0)    # Append time delta

 return min(time_hist)           # Return the min of the r deltas   
jedwards
  • 29,432
  • 3
  • 65
  • 92
  • 1
    Thanks for replying.. actually I did check the `timeit` module in `python` as well before posting.. and did see this.. What I was wondering is why it is necessary to have 2 different options `-n` and `-r`, why not just have 1? I will check the code that you mentioned in some more detail to see if there is any difference.. – alpha_989 Jun 16 '18 at 20:22
  • I played around with the function you mentioned.. A follow up question..When using just `-n`, the fundamental reason why you might get different values when running the same function/code, is because the underlying system may be behaving slightly differently from 1 run to another. To compensate for that we can use `-n` such that the total time is ~0.2s or higher. That makes sense.. This also takes care and accounts for the fact that `CPU may be scheduling other processes, or there may be network connections or you need to access the hard-disk which may be in use by something else` – alpha_989 Jun 17 '18 at 00:30
  • All these variabilities are accounted for by having a sufficiently large `-n` >=0.2s. As you mentioned `-r` determines how many repetitions of `-n` you are doing and its also taking care of the same variability (`CPU scheduling, Network communications, hard-disk access etc`). So if we used a sufficiently large `-r`, with `-n1`, wouldn't it just be doing the same thing? Because, fundamentally we are trying to make sure that we have sampled enough times, so that the variability in process time due to `CPU scheduling, network access, hard disk access` are accounted for.. – alpha_989 Jun 17 '18 at 00:30
  • Because regardless of whether you use `-n20 -r1`, or `-n1 -r20`, or just had 1 option which just provided an option to define the number of times the function is run, say `-l20`, they are all just doing random sampling, and capturing sufficient number of samples to have high signal/noise ratio? – alpha_989 Jun 17 '18 at 00:32