How to advance clock and going through all the events

Question

Reading this answer (point 2) to a question related to Twisted's task.Clock for testing purposes, I found very weird that there is no way to advance the clock from t0 to t1 while catching all the callLater calls within t0 and t1.

Of course, you could solve this problem by doing something like:

clock = task.Clock()
reactor.callLater = clock.callLater

...

def advance_clock(total_elapsed, step=0.01):
    elapsed = 0
    while elapsed < total_elapsed:
        clock.advance(step)
        elapsed += step

...

time_to_advance = 10  # seconds
advance_clock(time_to_advance)

But then we have shifted the problem toward choosing a sufficiently small step, which could be very tricky for callLater calls that sample the time from a probability distribution, for instance.

Can anybody think of a solution to this problem?

Answer 1

I found very weird that there is no way to advance the clock from t0 to t1 while catching all the callLater calls within t0 and t1.

Based on what you wrote later in your question, I'm going to suppose that the case you're pointing out is the one demonstrated by the following example program:

from twisted.internet.task import Clock

def foo(reactor, n):
    if n == 0:
        print "Done!"
    reactor.callLater(1, foo, reactor, n - 1)

reactor = Clock()
foo(reactor, 10)
reactor.advance(10)

One might expect this program to print Done! but it does not. If the last line is replaced with:

for i in range(10):
    reactor.advance(1)

Then the resulting program does print Done!.

The reason Clock works this way is that it's exactly the way real clocks work. As far as I know, there are no computer clocks that operate with a continuous time system. I won't say it is impossible to implement a timed-event system on top of a clock with discrete steps such that it appears to offer a continuous flow of time - but I will say that Twisted makes no attempt to do so.

The only real difference between Clock and the real reactor implementations is that with Clock you can make the time-steps much larger than you are likely to encounter in typical usage of a real reactor.

However, it's quite possible for a real reactor to get into a situation where a very large chunk of time all passes in one discrete step. This could be because the system clock changes (there's some discussion of making it possible to schedule events independent of the system clock so that this case goes away) or it could be because some application code blocked the reactor for a while (actually, application code always blocks the reactor! But in typical programs it only blocks it for a period of time short enough for most people to ignore).

Giving Clock a way to mimic these large steps makes it possible to write tests for what your program does when one of these cases arises. For example, perhaps you really care that, when the kernel decides not to schedule your program for 2.1 seconds because of a weird quirk in the Linux I/O elevator algorithm, your physics engine nevertheless computes 2.1 seconds of physics even though 420 calls of your 200Hz simulation loop have been skipped.

It might be fair to argue that the default (standard? only?) time-based testing tool offered by Twisted should be somewhat more friendly towards the common case... Or not. Maybe that would encourage people to write programs that only work in the common case and break in the real world when the uncommon (but, ultimately, inevitable) case arises. I'm not sure.

Regarding Mike's suggestion to advance exactly to the next scheduled call, you can do this easily and without hacking any internals. clock.advance(clock.getDelayedCalls()[0].getTime() - clock.seconds()) will do exactly this (perhaps you could argue Clock would be better if it at least offered an obvious helper function for this to ease testing of the common case). Just remember that real clocks do not advance like this so if your code has a certain desirable behavior in your unit tests when you use this trick, don't be fooled into thinking this means that same desirable behavior will exist in real usage.

Answer 2

Given that the typical use-class for Twisted is to mix hardware events and timers I'm confused why you would want to do this, but...

My understanding is that interally Twisted is tracking callLater events via a number of lists that are inside of the reactor object (See: http://twistedmatrix.com/trac/browser/tags/releases/twisted-15.2.0/twisted/internet/base.py#L437 - the xxxTimedCalls lists inside of class ReactorBase)

I haven't done any work to figure out if those lists are exposed anywhere, but if you want to take the reactors life into your own hands I'm sure you could hack your way in.

With access to the timing lists you could simply forward time to whenever the next element of the list is ... though if your trying to test code that interacts with IO events, I can't imagine this is going to do anything but confuse you...

Best of luck

Answer 3

Here's a function that will advance the reactor to the next IDelayedCall by iterating over reactor.getDelayedCalls. This has the problem Mike mentioned of not catching IO events, so you can specify a minimum and maximum time that it should wait, as well as a maximum time step.

def advance_through_delayeds(reactor, min_t=None, max_t=None, max_step=None):
    elapsed = 0
    while True:
        if max_t is not None and elapsed >= max_t:
            break
        try:
            step = min(d.getTime() - reactor.seconds() for d in reactor.getDelayedCalls())
        except ValueError:
            # nothing else pending
            if min_t is not None and elapsed < min_t:
                step = min_t - elapsed
            else:
                break
        if max_step is not None:
            step = min(step, max_step)
        if max_t is not None:
            step = min(step, max_t-elapsed)
        reactor.advance(step)
        elapsed += step
    return elapsed

If you need to wait for some I/O to complete, then set min_t and max_step to reasonable values.

# wait at least 10s, advancing the reactor by no more than 0.1s at a time
advance_through_delayeds(reactor, min_t=10, max_step=0.1)

If min_t is set, it will exit once getDelayedCalls returns an empty list after that time is reached.

It's probably a good idea to always set max_t to a sane value to prevent the test suite from hanging. For example, on the above foo function by JPC it does reach the print "Done!" statement, but then would hang forever as the callback chain never completes.