Copying a global table across Lua states

Question

I have a global table, which, I want to keep in sync, between two different Lua states. From what I have read, and understood, the only way seems to be, in my C back-end, do a deep copy of the table between the states (if the table has been modified). Is there a better way ? Also, I saw some Lua snippets for table deep copy, but not in C, are there any libraries which do this [in C]?

P.S. I am not looking for a lua_thread based solution (I am already using it)

P.P.S Lua Lanes seems to be close, but IMO, seems too much, because I just want to sync 1 table!

What about hooking some C code into [__newindex](http://lua-users.org/wiki/MetatableEvents) metatable key which would apply the same Lua table changes on an another thread. Doing deep copy just to sync single event sounds like a pornography in this case.. Also, iterating Lua table into depth on one Lua VM and recreating it on another - doesn't seam like a big challenge. Thought, i'd prefer some semaphores and serialization magics. — Kamiccolo, Oct 07 '13 at 19:16
If you use those states in different threads then you can not easy update state from different thread. You can write simple C module that support share data between states. It could use third lua_state to store real table or use some hash table or even some inproc database (LevelDB, SQLite, sophia - you can use existed modules if they thread safe). And each state should use this module to store/retrieve data through some proxy table. — moteus, Oct 08 '13 at 11:28
@moteus, the states are part of two `pthreads` in the same process. — Ani, Oct 08 '13 at 12:06
@Kamiccolo, I will try the `__newindex` approach (probably I will need serialization as well!) — Ani, Oct 08 '13 at 12:07

score 3 · Accepted Answer · answered Oct 08 '13 at 12:59

3

Note that __newindex won't work if the key already exists in the table you're writing to.

An alternative is to keep the table empty so that it never has any real contents. You can keep all the actual data in C, in which case neither state needs to have the table populated and your meta table can instead be used as a view onto your data from any thread. This has the bonus of not requiring a data copy on either side as the data will be available on request.

A custom __pairs function to iterate your internal data if required, plus an __index function to look at the data and you're away.

answered Oct 08 '13 at 12:59

Ian

433
3
6

oh ok, this seems like a nice idea, because anyway most of my data is in C, but I think I have to take extra care in this case (when access/writes happen from multiple `lua_States` [different `pthreads`]). Is there any sample code/gists for this on the web? – Ani Oct 08 '13 at 13:39
If you're using multiple states via lua_thread()/coroutine.create(), then you should define luaL_lock()/luaL_unlock(). There are examples of this on the web. Then in your C code, a simple mutex should be sufficient to control data access in your pseudo table. – Ian Oct 09 '13 at 08:16

score -1 · Answer 2 · answered Oct 08 '13 at 14:52

When you create Lua states, you have the option to pass in an allocator when you use lua_newstate (as against lua_open or luaL_newstate). Normally the allocator would just get a request similar to a realloc call. But you have the option to pass in a user defined pointer (the first arg) to the allocator.

You can pass the same allocator to two lua state create functions. Just before you create the global table(s) that you want to be shared, you just set the user defined pointer. Then you can return the reference to the same memory location to both the lua states. You wouldn't need any special code to share them. Sample code below:

static char giant_shared_block[1000000]; 
static void *l_alloc (void *ud, void *ptr, size_t osize, size_t nsize) {    
    if (nsize == 0) {
        if(ptr != giant_shared_block && osize != 0)
            free(ptr);
        return NULL;
    }
    else{
        int is_shared = *((int *)ud);
        if(is_shared){ //indicated by the user prior to creating global memory block in a lua state
            *ud = 0; //unset the flag
            return giant_shared_block;
        }
        else if(osize == 0)
            return malloc(nsize);
        else
            return realloc(ptr, nsize);
    }
}

Of course the onus on the user to make sure to set the 'shared' flag before creating the table. The first memory allocation after the flag is set, will be shared.

I haven't seen this pattern before .., but updates from multiple threads still needs to be synchronised. I'll keep this approach also, as an open option. — Ani, Oct 08 '13 at 18:00
That's the point. There's no 'synchronization' for the shared global table. There's only one of them. — Arun R, Oct 08 '13 at 18:07
I doubt very much that would work. As soon as one state writes a string or other chunk to the table, the other would not be able to access it being in a different root state. And I would expect the GC system to have some issues with it too, with objects belonging to multiple states. — Ian, Oct 09 '13 at 08:23

Copying a global table across Lua states

2 Answers2