26

I ran down a bug today that came about because I was using next() to extract a value, and 'not found' emits a StopIteration.

Normally that would halt the program, but the function using next was being called inside an all() iteration, so the all just terminated early and returned True.

Is this an expected behavior? Are there style guides that help avoid this kind of thing?

Simplified example:

def error(): return next(i for i in range(3) if i==10)
error() # fails with StopIteration
all(error() for i in range(2)) # returns True
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
amwinter
  • 3,121
  • 2
  • 27
  • 25
  • The same thing happens in Python 3. – khelwood Feb 02 '15 at 23:43
  • @khelwood thanks, I'll remove the py2.7 tag – amwinter Feb 02 '15 at 23:44
  • 9
    Of course. [**all** *(iterable)* Return True if all elements of the iterable are true ***(or if the iterable is empty)***.](https://docs.python.org/2/library/functions.html#all) – Frédéric Hamidi Feb 02 '15 at 23:44
  • `iter([])` could replace your function. Interestingly, `any(iter([]))` is False. – tdelaney Feb 02 '15 at 23:48
  • 5
    @tdelaney because `any` returns `True` if the iterable contains any value that is `True`. `all` returns `True` if the iterable contains NO values that are `False`. – Adam Smith Feb 02 '15 at 23:49
  • 2
    So..., all items in the collection are truthy (because there aren't any) but no single item is truthy (because there aren't any). – tdelaney Feb 02 '15 at 23:50
  • 1
    `next((i for i in range(3) if i==10), None)` will return `None` instead of raising a `StopIteration` exception. – Peter Wood Feb 02 '15 at 23:50
  • 1
    @AdamSmith - you stole my line! I should have typed faster. – tdelaney Feb 02 '15 at 23:51
  • @tdelaney: `any` means `at least 1`. All means `every single one`. – njzk2 Feb 02 '15 at 23:51
  • @tdelaney, *all items in the collection are truthy (because there aren't any)*. It doesn't work like that. The *collection* satisfies the `all()` predicate because it is empty, so no items violate the predicate. Conversely, the collection does not satisfy the `any()` predicate because it is empty, so no single item satisfies that predicate. – Frédéric Hamidi Feb 02 '15 at 23:54
  • 3
    @FrédéricHamidi - um, that's what i said. – tdelaney Feb 02 '15 at 23:55
  • @tdelaney, um, I may have misunderstood, but you said *all items in the collection are truthy*, and I said that's not what we mean in the general case. – Frédéric Hamidi Feb 02 '15 at 23:56
  • 1
    @FredericHamidi "all items are truthy" is *exactly equivalent to* "there are no items which are not truthy" – user253751 Feb 03 '15 at 08:00

2 Answers2

24

While this is the default behaviour in Python versions up to and including 3.6, it's considered to be a mistake in the language, and is scheduled to change in Python 3.7 so that an exception is raised instead.

As PEP 479 says:

The interaction of generators and StopIteration is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, StopIteration raised accidentally inside a generator function will be interpreted as the end of the iteration by the loop construct driving the generator.

From Python 3.5 onwards, it's possible to change the default behaviour to that scheduled for 3.7. This code:

# gs_exc.py

from __future__ import generator_stop

def error():
    return next(i for i in range(3) if i==10)

all(error() for i in range(2))

… raises the following exception:

Traceback (most recent call last):
  File "gs_exc.py", line 8, in <genexpr>
    all(error() for i in range(2))
  File "gs_exc.py", line 6, in error
    return next(i for i in range(3) if i==10)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "gs_exc.py", line 8, in <module>
    all(error() for i in range(2))
RuntimeError: generator raised StopIteration

In Python 3.5 and 3.6 without the __future__ import, a warning is raised. For example:

# gs_warn.py

def error():
    return next(i for i in range(3) if i==10)

all(error() for i in range(2))


$ python3.5 -Wd gs_warn.py 
gs_warn.py:6: PendingDeprecationWarning: generator '<genexpr>' raised StopIteration
  all(error() for i in range(2))


$ python3.6 -Wd gs_warn.py 
gs_warn.py:6: DeprecationWarning: generator '<genexpr>' raised StopIteration
  all(error() for i in range(2))
Zero Piraeus
  • 56,143
  • 27
  • 150
  • 160
9

The problem isn't in using all, it's that you have a generator expression as the parameter to all. The StopIteration gets propagated to the generator expression, which doesn't really know where it originated, so it does the usual thing and ends the iteration.

You can see this by replacing your error function with something that raises the error directly:

def error2(): raise StopIteration

>>> all(error2() for i in range(2))
True

The final piece of the puzzle is knowing what all does with an empty sequence:

>>> all([])
True

If you're going to use next directly, you should be prepared to catch StopIteration yourself.

Edit: Nice to see that the Python developers consider this a bug and are taking steps to change it in 3.7.

Mark Ransom
  • 299,747
  • 42
  • 398
  • 622
  • 1
    ok -- so you're saying the solution is to not use `any`/`all` around anything that can raise a `StopIteration`? – amwinter Feb 02 '15 at 23:56
  • @PadraicCunningham it's caught by the generator expression itself. – Mark Ransom Feb 02 '15 at 23:57
  • OP can either catch the exception or use the default value of `next` – jamylak Feb 02 '15 at 23:59
  • @amwinter it's more than that - the advice is to *not* call anything that could raise `StopIteration` from within a generator expression. – Mark Ransom Feb 02 '15 at 23:59
  • Not sure why this was downvoted - it's a good explanation of the current state of affairs. – Zero Piraeus Feb 03 '15 at 00:44
  • It was me that downvoted, but it was a total accident... I must have accidentally clicked my mouse when the cursor was on the downvote button. I just noticed and it won't let me undo because too much time has expired. @MarkRansom if you make an edit I can undo my accidental downvote. – SethMMorton Feb 03 '15 at 00:58