How to iterate over a table modified with luaL_ref and luaL_unref?

Question

I'm using Lua's C API to extend Lua. In my module, I want to populate a table using luaL_ref, and remove fields using luaL_unref. I also want to be able to iterate over this table, hopefully using lua_next.

Iterating over the table is a problem because of luaL_unref. In Lua it is common to "delete" table fields by assigning nil (because uninitialized table fields evaluate to nil). The next function is smart enough to skip over nil. I would have expected luaL_unref to assign nil to unreferenced table fields, but it seems to assign an integer. The value of this integer seems to be undocumented.

Consider the following code:

/* tableDump prints a table: */
/* key: value, key: value, ... */

lua_newtable(L);

lua_pushboolean(L, 0);
int ref1 = luaL_ref(L, -2);

lua_pushinteger(L, 7);
int ref2 = luaL_ref(L, -2);

lua_pushstring(L, "test");
int ref3 = luaL_ref(L, -2);

tableDump(L, -1);

luaL_unref(L, -1, ref1);
tableDump(L, -1);

luaL_unref(L, -1, ref3);
tableDump(L, -1);

luaL_unref(L, -1, ref2);
    tableDump(L, -1);

printf("done.\n");

Output:

1:  false,  2:  7,  3:  `test',  
3:  `test',  2:  7,  0:  1,  
3:  1,  2:  7,  0:  3,  
3:  1,  2:  3,  0:  2,  
done.

What's going on here? How could I work around this? Is there some trick to iterate over references and ignore the unreferenced? Do I have to stop using luaL_ref and luaL_unref?

Edit

First off, thank you for your responses!

Maybe I've asked the wrong question.

Allow me to be a little more specific. I have a client userdata which needs to manage many subscription userdatas. Subscriptions are created by the client's subscribe method. Subscriptions are removed by the client's unsubscribe method. The subscription userdatas are basically an implementation detail, so they are not exposed in the client API. Instead the client API uses subscription references, hence the use of luaL_ref to populate a subscription table.

ref = client:sub(channel, func)
cleint:unsub(ref)

Here's the catch. I would like the client to automatically unsubscribe all remaining subscriptions on __gc (or else the user will get a segfault). So it seems I need to iterate over the subscriptions. Am I really abusing the API here? Is there a better way to do this?

Stop abusing the API. The purpose of `luaL_ref` and `luaL_unref` is *not* to "populate a table". It is to create a simple reference system for being able to store values by reference and fetch them again later. You should *not* be iterating over the table; you should be using the returned reference to access table entries. — Nicol Bolas, Oct 07 '12 at 19:18

score 3 · Accepted Answer · answered Oct 09 '12 at 01:37

The luaL_ref and luaL_unref functions are defined in lauxlib.c. They work by keeping track of a free reference list, which they store in the table they are operating on. These functions are relatively short, so I will include them here.

LUALIB_API int luaL_ref (lua_State *L, int t) {
  int ref;
  t = abs_index(L, t);
  if (lua_isnil(L, -1)) {
    lua_pop(L, 1);  /* remove from stack */
    return LUA_REFNIL;  /* 'nil' has a unique fixed reference */
  }
  lua_rawgeti(L, t, FREELIST_REF);  /* get first free element */
  ref = (int)lua_tointeger(L, -1);  /* ref = t[FREELIST_REF] */
  lua_pop(L, 1);  /* remove it from stack */
  if (ref != 0) {  /* any free element? */
    lua_rawgeti(L, t, ref);  /* remove it from list */
    lua_rawseti(L, t, FREELIST_REF);  /* (t[FREELIST_REF] = t[ref]) */
  }
  else {  /* no free elements */
    ref = (int)lua_objlen(L, t);
    ref++;  /* create new reference */
  }
  lua_rawseti(L, t, ref);
  return ref;
}

LUALIB_API void luaL_unref (lua_State *L, int t, int ref) {
  if (ref >= 0) {
    t = abs_index(L, t);
    lua_rawgeti(L, t, FREELIST_REF);
    lua_rawseti(L, t, ref);  /* t[ref] = t[FREELIST_REF] */
    lua_pushinteger(L, ref);
    lua_rawseti(L, t, FREELIST_REF);  /* t[FREELIST_REF] = ref */
  }
}

These functions are really clever because they require no additional storage (besides the table they are operating on). However, it is occasionally undesirable to have the free reference list stored in the same table as the referenced values, such as when it is necessary to iterate over referenced values.

I've written luaX_ref and luaX_unref to solve this problem. They work nearly the same as luaL_ref and luaL_unref, except they store their free reference list in a separate table, l for list. (For my project, I've put these in a separate source file and have included them as necessary. I do not recommend modifying lauxlib.c.)

static int abs_index(lua_State * L, int i){
  return i > 0 || i <= LUA_REGISTRYINDEX ? i : lua_gettop(L) + i + 1;
}

LUALIB_API int luaX_ref(lua_State *L, int t, int l){
  int ref;
  t = abs_index(L, t);
  l = abs_index(L, l);
  if(lua_isnil(L, -1)){
    lua_pop(L, 1); /* remove from stack */
    return LUA_REFNIL; /* 'nil' has a unique fixed reference */
  }
  lua_rawgeti(L, l, FREELIST_REF); /* get first free element */
  ref = (int) lua_tointeger(L, -1); /* ref = l[FREELIST_REF] */
  lua_pop(L, 1); /* remove it from stack */
  if(ref != 0){ /* any free element? */
    lua_rawgeti(L, l, ref); /* remove it from list */
    lua_rawseti(L, l, FREELIST_REF); /* (l[FREELIST_REF] = l[ref]) */
  }else{ /* no free elements */
    ref = (int)lua_objlen(L, l);
    ref++; /* create new reference */
  }
  lua_pushboolean(L, 1);
  lua_rawseti(L, l, ref); /* l[ref] = true */
  lua_rawseti(L, t, ref); /* t[ref] = value */
  return ref;
}

LUALIB_API void luaX_unref(lua_State *L, int t, int l, int ref){
  if(ref >= 0){
    t = abs_index(L, t);
    l = abs_index(L, l);
    lua_rawgeti(L, l, FREELIST_REF);
    lua_rawseti(L, l, ref);  /* l[ref] = l[FREELIST_REF] */
    lua_pushinteger(L, ref);
    lua_rawseti(L, l, FREELIST_REF);  /* l[FREELIST_REF] = ref */
    lua_pushnil(L);
    lua_rawseti(L, t, ref); /* t[ref] = nil */
  }
}

Now see the usage:

lua_newtable(L); /* 1 */
lua_newtable(L); /* 2 */

lua_pushboolean(L, 0);
int ref1 = luaX_ref(L, 1, 2);

lua_pushinteger(L, 7);
int ref2 = luaX_ref(L, 1, 2);

lua_pushstring(L, "test");
int ref3 = luaX_ref(L, 1, 2);

tableDump(L, 1);
tableDump(L, 2);

luaX_unref(L, 1, 2, ref1);
tableDump(L, 1);
tableDump(L, 2);

luaX_unref(L, 1, 2, ref3);
tableDump(L, 1);
tableDump(L, 2);

luaX_unref(L, 1, 2, ref2);
tableDump(L, 1);
tableDump(L, 2);

printf("done.\n");

Output:

1:  false,  2:  7,  3:  `test',
1:  true,  2:  true,  3:  true,
2:  7,  3:  `test',
3:  true,  2:  true,  0:  1,
2:  7,
3:  1,  2:  true,  0:  3,

3:  1,  2:  3,  0:  2,
done.

score 2 · Answer 2 · answered Oct 07 '12 at 19:41

In order to "ensure the uniqueness of the key it returns" luaL_ref must maintain a list of used and subsequently luaL_unref removed keys. It appears that this list begins at t[0] and continues until the chain of indices leads to a nil. This list is kept in the same table as the active references.

If you want to continue to "abuse the API" as Nicol observed, and depend on implementation defined behavior, you could follow this linked list to see if a key is a removed reference as you iterate the table. Or to avoid the dependence on implementation defined behavior, and be more performant, you could keep a separate table of removed references and skip them as you iterate the table, though you'll need to ignore the list head at t[0].

If you really need to iterate the references you may be better off using a different mechanism altogether. You could simply put all your reference/value pairs in a separate table, and set the value in the separate table to nil when you remove the reference. Then you can simply iterate the separate table.

How to iterate over a table modified with luaL_ref and luaL_unref?

2 Answers2