52

I am trying to iterate the lambda func over a list as in test.py, and I want to get the call result of the lambda, not the function object itself. However, the following output really confused me.

------test.py---------
#!/bin/env python
#coding: utf-8

a = [lambda: i for i in range(5)]
for i in a:
    print i()

--------output---------
<function <lambda> at 0x7f489e542e60>
<function <lambda> at 0x7f489e542ed8>
<function <lambda> at 0x7f489e542f50>
<function <lambda> at 0x7f489e54a050>
<function <lambda> at 0x7f489e54a0c8>

I modified the variable name when print the call result to t as following, and everything goes well. I am wondering what is all about of that. ?

--------test.py(update)--------
a = [lambda: i for i in range(5)]
for t in a:
    print t()

-----------output-------------
4
4
4
4
4

This behaviour is 2.x-specific, and is a special case of the problem described at What do lambda function closures capture?. In 3.x, the list comprehension creates its own scope for the iteration variable, so it isn't modifiable from outside. It is, however, still late binding, so each print will produce 4.

Karl Knechtel
  • 62,466
  • 11
  • 102
  • 153
keyuan7569
  • 443
  • 4
  • 6
  • 1
    Sorry for closing, I misread the question. – Daniel Roseman Jul 14 '16 at 09:03
  • 1
    Whoops, sorry for flagging. I misread as well! – juanpa.arrivillaga Jul 14 '16 at 09:28
  • 3
    I changed the title because it was *really* easy to skim your question and think you had a simple late-binding problem instead of a problem related to reusing the variable name. (This is evidenced in the mistaken closures and answers.) The new title calls attention to the real issue you're concerned about. If you want to reword it from what I've made it, please make sure you preserve that aspect. – jpmc26 Jul 14 '16 at 16:02
  • 4
    The above demonstration of civility and humility is what I love about the Python community, especially those Pythonistas who frequent SO. – Wayne Conrad Jul 14 '16 at 17:14

3 Answers3

46

In Python 2 list comprehension 'leaks' the variables to outer scope:

>>> [i for i in xrange(3)]
[0, 1, 2]
>>> i
2

Note that the behavior is different on Python 3:

>>> [i for i in range(3)]
[0, 1, 2]
>>> i
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'i' is not defined

When you define lambda it's bound to variable i, not its' current value as your second example shows. Now when you assign new value to i the lambda will return whatever is the current value:

>>> a = [lambda: i for i in range(5)]
>>> a[0]()
4
>>> i = 'foobar'
>>> a[0]()
'foobar'

Since the value of i within the loop is the lambda itself you'll get it as a return value:

>>> i = a[0]
>>> i()
<function <lambda> at 0x01D689F0>
>>> i()()()()
<function <lambda> at 0x01D689F0>

UPDATE: Example on Python 2.7:

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> a = [lambda: i for i in range(5)]
>>> for i in a:
...     print i()
... 
<function <lambda> at 0x7f1eae7f15f0>
<function <lambda> at 0x7f1eae7f1668>
<function <lambda> at 0x7f1eae7f16e0>
<function <lambda> at 0x7f1eae7f1758>
<function <lambda> at 0x7f1eae7f17d0>

Same on Python 3.4:

Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = [lambda: i for i in range(5)]
>>> for i in a:
...     print(i())
... 
4
4
4
4
4

For details about the change regarding the variable scope with list comprehension see Guido's blogpost from 2010.

We also made another change in Python 3, to improve equivalence between list comprehensions and generator expressions. In Python 2, the list comprehension "leaks" the loop control variable into the surrounding scope:

x = 'before'
a = [x for x in 1, 2, 3]
print x # this prints '3', not 'before'

However, in Python 3, we decided to fix the "dirty little secret" of list comprehensions by using the same implementation strategy as for generator expressions. Thus, in Python 3, the above example (after modification to use print(x) :-) will print 'before', proving that the 'x' in the list comprehension temporarily shadows but does not override the 'x' in the surrounding scope.

niemmi
  • 17,113
  • 7
  • 35
  • 42
  • 2
    The leaky scope of list comprehensions in python 2 doesn't affect the behavior in the OP. – juanpa.arrivillaga Jul 14 '16 at 09:23
  • 2
    @juanpa.arrivillaga Have you tried running the code in question with both versions? At least I get different results with Python 2.7 & 3.4. – niemmi Jul 14 '16 at 09:25
  • Whoops! I totally misread this question! You're totally right! This is surprising behavior to me... – juanpa.arrivillaga Jul 14 '16 at 09:29
  • 1
    On second thought, this is *not* surprising behavior at all. – juanpa.arrivillaga Jul 14 '16 at 09:36
  • Thanks very much for your detailed explanation, and I got it! ;) – keyuan7569 Jul 14 '16 at 09:43
  • @keyuan7569 I'm happy to help and learn something myself as well :) – niemmi Jul 14 '16 at 09:52
  • @niemmi, there is kind of confusing for me, when I modified the code to for i in a:print i()()()()(), it still outputs the function object itself. Is it a unlimit recursive, and is that behavior reasonable? – keyuan7569 Jul 14 '16 at 10:04
  • @keyuan7569 Well the lambda returns itself which in turn returns itself and so forth so you have infinite recursion. Whether it's reasonable or not is a matter of opinion. – niemmi Jul 14 '16 at 11:21
  • @juanpa.arrivillaga "On second thought, this is not surprising behavior at all." If your first thought was different, that's kind of the definition of surprising behaviour :) – Tavian Barnes Jul 14 '16 at 18:47
  • The same scope leak is the cause of [List comprehension rebinds names even after scope of comprehension. Is this right?](https://stackoverflow.com/questions/4198906). – Karl Knechtel Aug 19 '22 at 12:25
19

Closures in Python are late-binding, meaning that each lambda function in the list will only evaluate the variable i when invoked, and not when defined. That's why all functions return the same value, i.e. the last value of ì (which is 4).

To avoid this, one technique is to bind the value of i to a local named parameter:

>>> a = [lambda i=i: i for i in range(5)]
>>> for t in a:
...   print t()
... 
0
1
2
3
4

Another option is to create a partial function and bind the current value of i as its parameter:

>>> from functools import partial
>>> a = [partial(lambda x: x, i) for i in range(5)]
>>> for t in a:
...   print t()
... 
0
1
2
3
4

Edit: Sorry, misread the question initially, since these kind of questions are so often about late binding (thanks @soon for the comment).

The second reason for the behavior is list comprehension's variable leaking in Python2 as others have already explained. When using i as the iteration variable in the for loop, each function prints the current value of i (for the reasons stated above), which is simply the function itself. When using a different name (e.g. t), functions print the last value of i as it was in the list comprehension loop, which is 4.

Community
  • 1
  • 1
plamut
  • 3,085
  • 10
  • 29
  • 40
  • Well, but why changing the variable name in the `for` loop changes the output? – awesoon Jul 14 '16 at 09:14
  • @soon What variable are you referring to, the `t`? That does not affect anything, it's just a temporary name we give to each function when iterating over the list. Or did I misunderstand the question? – plamut Jul 14 '16 at 09:16
  • 1
    My interpretation of the question was "why `for i in a` prints lambdas, but `for k in a` prints numbers?". Of course, this is because of late-binding too, but this may be unclear for the OP. +1 anyway – awesoon Jul 14 '16 at 09:26
  • Ah, I see, I may have jumped into the late binding behavior explanation too hastily. :) It's because of the variable leaking in Python 2 as others have already explained. – plamut Jul 14 '16 at 09:32
4

lambda: i is an anonymous function with no arguments that returns i. So you are generating a list of anonymous functions, which you can later (in the second example) bind to the name t and invoke with (). Note you can do the same with non-anonymous functions:

>>> def f():
...   return 42
... 
>>> name = f
>>> name
<function f at 0x7fed4d70fb90>
>>> name()
42

@plamut has just answered the implied other part of the question, so I won't.

nigel222
  • 7,582
  • 1
  • 14
  • 22