python closure with assigning outer variable inside inner function

Question

I've got this piece of code:

#!/usr/bin/env python

def get_match():
  cache=[]
  def match(v):
    if cache:
      return cache
    cache=[v]
    return cache
  return match
m = get_match()
m(1)

if I run it, it says:

UnboundLocalError: local variable 'cache' referenced before assignment

but if I do this:

#!/usr/bin/env python

def get():
  y = 1
  def m(v):
    return y + v
  return m

a=get()
a(1)

it runs.

Is there something with list? or my code organizing is wrong?

Answer 1

The problem is that the variable cache is not in the scope of the function match. This is not a problem if you only want to read it as in your second example, but if you're assigning to it, python interprets it as a local variable. If you're using python 3 you can use the nonlocal keyword to solve this problem - for python 2 there's no simple workaround, unfortunately.

def f():
    v = 0

    def x():
        return v    #works because v is read from the outer scope

    def y():
        if v == 0:  #fails because the variable v is assigned to below
            v = 1

    #for python3:
    def z():
        nonlocal v  #tell python to search for v in the surrounding scope(s)
        if v == 0:
            v = 1   #works because you declared the variable as nonlocal

The problem is somewhat the same with global variables - you need to use global every time you assign to a global variable, but not for reading it.

A short explanation of the reasons behind that: The python interpreter compiles all functions into a special object of type function. During this compilation, it checks for all local variables the function creates (for garbage collection etc). These variable names are saved within the function object. As it is perfectly legal to "shadow" an outer scopes variable (create a variable with the same name), any variable that is assigned to and that is not explicitly declared as global (or nonlocal in python3) is assumed to be a local variable.

When the function is executed, the interpreter has to look up every variable reference it encounters. If the variable was found to be local during compilation, it is searched in the functions f_locals dictionary. If it has not been assigned to yet, this raises the exception you encountered. If the variable is not assigned to in the functions scope and thus is not part of its locals, it is looked up in the surrounding scopes - if it is not found there, this raises a similar exception.

Answer 2

Accessing a variable is different from assigning it.

You have a similar situation with global variables. You can access them in any function, but if you try to assign to it without the global statement, it will redeclare it in the local context.

Unfortunately for local functions there is no equivalent of the global statement, but you can bypass the redeclaration by replacing

cache=[v]

with:

cache[:] = [v]

Answer 3

Since Python sees cache=[v] - assignment to cache, it treats it as local variable. So the error is pretty reasonable - no local variable cache was defined prior to its usage in if statement.

You probably want to write it as:

def get_match():
  cache=[]
  def match(v):
    if cache:
      return cache
    cache.append(v)
    return cache
  return match
m = get_match()
m(1)

Highly recommended readings: Execution Model - Naming and binding and PEP 227 - Statically Nested Scopes

Answer 4

Replace

cache=[]
def match(v):

with

def match(v,cache=[])

Explanation: Your code declares cache as a variable of get_match, which the returned match(v) knows nothing about (due to the following assignment). You do however want cache to be part of match's namespace.

I know this way a "malevolent" user could redefine cache, but that's their own mistake. Should this be an issue though, the alternative is:

def match(v):
     try:
         if cache:
             return cache
     except NameError:
         cache = []
     ...

(See here)