Lua upvalues not behaving as expected with local variables

Question

Consider this Lua 5.1 code:

function foo ()
 function makeAdder (withWhat)
   return function (a)
     return a + withWhat
   end
 end -- makeAdder
 f1 = makeAdder (6)
 f2 = makeAdder (7)
end -- for

foo ()
print (f1 (2))  --> 8
print (f2 (2))  --> 9

The functions f1 and f2 have 6 and 7 as upvalues to the closures, and thus output 8 and 9 in response to being called. This is what I expect.

Now consider:

function foo ()
 local withWhat
 function makeAdder ()
   return function (a)
     return a + withWhat
   end
 end -- makeAdder

 withWhat = 6
 f1 = makeAdder ()
 withWhat = 7
 f2 = makeAdder ()
end -- for

foo ()
print (f1 (2))  --> 9
print (f2 (2))  --> 9

The functions f1 and f2 have 6 and 7 as upvalues to the closures at the time they are created, however they output 9 and 9, which is not what I expect.

I thought f1 and f2 would be closed with their upvalues at the point of their creation, but obviously not. How does 7 become an upvalue for f1, even though it "should" have been 6?

Lua version 5.1.5.

Luatic · Answer 1 · 2023-08-05T09:42:05.530

In Lua, upvalues are variables. The reference manual refers to them as "external local variables". They aren't constants that are copied once at the "closing" of the function; they are mutable variables. The non-local variables are bound to storage locations rather than to values. A more minimal example to demonstrate this behavior would be:

local upval = 42
function f() return upval end -- upval is not copied, only referenced
upval = 33 -- the upvalue is a reference to a value...
-- ... so reassigning it modifies the upvalue `f` operates with:
print(f()) -- 33

In your example, the functions returned inside foo by makeAdder all share the same withWhat upvalue, so modifying it changes the upvalue the adders operate with. To explicitly copy at the time of "closing" f1 and f2, assign to a local variable inside makeAdder:

function makeAdder()
   local withWhat = withWhat
   return function(a)
       return a + withWhat
   end
end

Side notes: Lua does not suffer the problem of Go et al. that loop variables are common to all anonymous functions in the loop. That is

local fs = {}
for i = 1, 3 do fs[i] = function() print(i) end end
for i = 1, 3 do fs[i]() end

will print 1 to 3 as expected (rather than printing 3 as using local i = 1; while i <= 3 do fs[i] = function() print(i) end; i = i + 1 end would), since for var = ... do ... end (and also for-in) creates a local variable scope inside the loop. In practice you can even modify the loop variable without affecting the loop (though this is not guaranteed).

The fact that upvalues are mutable and not copied at closure creation is very useful; without it, corecursion would require other means of indirection, such as global/environmental variables:

local bar -- "forward declaration" of function bar as upvalue to foo
function foo()
    print("foo")
    return bar()
end
-- This has to change the upvalue `foo` uses for the corecursion to work.
function bar()
    print("bar")
    return foo()
end
foo() -- start infinite alternating printing of foo and bar

besides, it allows making a simple, inefficient form of OOP where all methods can treat the common upvalues as a set of shared "instance variables":

function newFooBar()
    local foo = "foo"
    local bar = "bar"
    return {
        toggle = function() foo, bar = bar, foo end,
        print = function() print(foo, bar) end,
    }
end
local foobar = newFooBar()
foobar.print() -- foo bar
foobar.toggle()
foobar.print() -- bar foo

This is an excellent answer (which I voted for) however it doesn't explain why functions f1 and f2 in my example return different results in my first block of code. I'm going to accept shingo's answer because they explained about creating an upvalue if it doesn't already exist. I'm assuming that a function argument doesn't exist as an upvalue, which is why it has to be created in this case. — Nick Gammon, Aug 05 '23 at 22:34
Or am I using the wrong terminology? The function argument is effectively a local variable and not an upvalue? — Nick Gammon, Aug 05 '23 at 22:37
Local variables become upvalues to closures created in their scope; in `local x = 42; function f() return x end`, `x` becomes an upvalue of `f` when `f` is instantiated. Of course, this can cross multiple function scopes, as in `local x = 42; function f() return function() return x end end; g = f()`. shingo's answer goes into the details of how these two cases are implemented in the reference implementation. — Luatic, Aug 05 '23 at 23:02

shingo · Accepted Answer · 2023-08-05T12:04:08.217

Upvalues of a inner function in lua are copied by address from the enclosing function, resulting in both f1 and f2's withWhat upvalues in your second code pointing to the same address.

There are actually two ways to copy:

If an upvalue of the inner function refers to an upvalue of the enclosing function, lua simply copies the pointer.
```
ncl->l.upvals[j] = cl->upvals[GETARG_B(*pc)];
```
This is the case with your second piece of code. As for both makeAdder and function(a), withWhat is their upvalue, so the withWhat variable in both functions point to the same address.
If an upvalue of the inner function refers to a local variable of the enclosing function, lua attempts to find the corresponding upvalue from the list of created upvalues in the current stack, it will create a new one if the find fails. (See luaF_findupval)
```
ncl->l.upvals[j] = luaF_findupval(L, base + GETARG_B(*pc));
```
This is the case with your first piece of code. withWhat is a variable of the makeAdder function and an upvalue of the function(a) function, so lua will create a new upvalue for the function(a) function to store withWhat.

If you have another function inside the makeAdder function that uses the withWhat variable, lua will reuses the created upvalue. Consider the code:
```
function foo ()
 function makeAdder (withWhat)
   return function (a)
     local b = a + withWhat
     withWhat = withWhat + 10
     return b
   end, function (a)
     return a + withWhat --Will be affected by the above function
   end
 end

 f1, f2 = makeAdder (6)
end

foo ()
print (f1 (2))  --> 8
print (f2 (2))  --> 18
```

Lua upvalues not behaving as expected with local variables

2 Answers2