Not able to modify C++ object passed as parameter to Lua function

Question

I have been working on SDL2 2D Game Engine for several years now. Just ditched inheritance approach to define game entities with composition approach where I have Entity class and it has vector of Component classes and recently I got into lua, because I want to define entities using Lua table with optional callback functions.

Working parts

I am using Lua5.4 and C API to bind some engine methods and Entity class to Lua. I use XML file to load list of scripts for each Entity defined by Lua:

  <script name="player" filename="scripts/player.lua" type="entity"/>

Then Entity gets created in C++ with ScriptComponent which holds a pointer to Lua state. Lua file gets loaded at this point and state is not closed unless Entity is destroyed. player.lua script might look something like this:

  -- Entity
player = {
    -- Entity components
    transform = {
         X = 100,
         Y = 250
    },
    physics = {
        mass = 1.0,
        friction = 0.2
    },
    sprite = {
        id = "player",
        animation = {},
        width = 48,
        height = 48
    },
    collider = {
        type = "player",
        onCollide = function(this, second)
            print("Lua: onCollide() listener called!")
        end
    },
    HP = 100
}

Using this I managed to create each Component class using Lua C API with no issues. Also while loading this I detect and set "onCollide" function in Lua.

Also I have managed to register some Engine functions so I can call them to lua: playSound("jump") in C++:

static int lua_playSound(lua_State *L) {
    std::string soundID = (std::string)lua_tostring(L, 1);
    TheSoundManager::Instance()->playSound(soundID, 0);
    return 0;
}

Also have created meta table for Entity class with __index and __gc metamethods and it works if I call these methods with Entity created in Lua outside of player table, such as:

-- This goes in player.lua script after the main table
testEntity = Entity.create() -- works fine, but entity is created in Lua
testEntity:move(400, 400)
testEntity:scale(2, 2)
testEntity:addSprite("slime", "assets/sprite/slime.png", 32, 32)

Problem

Now whenever collision happens and Entity has ScriptComponent, it correctly calls onCollide method in Lua. Even playSound method inside triggers correctly. Problem is when I try to manipulate Entities which are passed as this and seconds arguments to onCollide

onCollide = function(this, second)
        print(type(this)) -- userdata
        print(type(second)) --userdata
        --Entity.scale(this, 10, 10) --segfault
        --this:scale(10, 10) --segfault
        playSound("jump") -- works fine, does not need any metatables
    end

This is how I am calling onCollide method and passing existing C++ object to Lua:

// This is found in a method which belongs to ScriptComponent class, it holds lua state
// owner is Entity*, all Components have this
// second is also Entity*
if (lua_isfunction(state, -1)) {
    void* self = (Entity*)lua_newuserdata(state, sizeof(Entity));
    self = owner;

    luaL_getmetatable(state, "EntityMetaTable");
    assert(lua_isuserdata(state, -2));
    assert(lua_istable(state, -1));
    lua_setmetatable(state, -2);
    assert(lua_isuserdata(state, -1));

    void* second = (Entity*)lua_newuserdata(state, sizeof(Entity));
    second = entity;
                            
    luaL_getmetatable(state, "EntityMetaTable");
    lua_setmetatable(state, -2);

    // Code always reaches cout statement below unless I try to manipulate Entity
    // objects passed to Lua in Lua                     
    if (luaOk(state, lua_pcall(state, 2, 0, 0))) {
        std::cout << "onCollide() Called sucessfully!!!" << std::endl;
    }
    script->clean(); // Cleans lua stack
    return;

}

So basically I have managed to load data from table, bind and use some methods from C++ engine and mapped Entity class using metatable and __index and __gc meta methods which work fine for objects created in Lua but not when I try to pass existing C++ object and set existing meta table.

I still think I will be alright without using any Lua binders, because all I wanted here is to load data for all Components which works fine and script some behaviour based on events which also almost works except for not being able to correctly pass existing C++ object to onCollide method. Thank you for your help!

At quick glance: you are allocating userdata from lua with `lua_newuserdata` correctly, but then you overwrite the pointer with whichever one you had before. The lines `self = owner;` and `second = entity;` don't copy any data, and you lose the original pointers. Do you have some memory management that you could use, i.e. `std::shared_ptr` instead of `Entity*`? — IS4, Dec 11 '22 at 12:18
I do have EntityManager class. It holds std::vector of std::unique_ptr. Do they need to be std::shared_ptr instead? So basically I need to pass Entities as smart pointers instead and then I can do self=owner? — kaktusas2598, Dec 11 '22 at 12:38
You could do that, but it is still a bit more complex. Answer incoming ;-) — IS4, Dec 11 '22 at 12:40
This might be a bit problematic because each Component class uses Component(Entity* owner) constructor. So when ScriptComponent gets added to Entity, it holds raw pointer t o it. It works perfectly on all other components, like Physics, Sprite and so on, so I am thinking maybe I don't even need ScriptComponent class and instead some kind of other mechanism to handle lua scripts in engine — kaktusas2598, Dec 11 '22 at 12:44

score 1 · Accepted Answer · answered Dec 11 '22 at 12:49

It is impossible to "pass an existing C++ object to Lua". You would either have to copy it to the storage you get with lua_newuserdata, or use lua_newuserdata from the beginning.

Depending on what you can guarantee about the lifetime of the entity, you have several options:

If the lifetime of every entity aligns with the lifetime of the Lua instance, you can let Lua manage the entity object completely through its GC, i.e. call lua_newuserdata and use the "placement new" operator on the resulting object, initializing the entity. Note however that you have to prevent the object from getting freed by the Lua GC somehow if you only use it outside Lua, by adding it to the registry table for example.
If the lifetime of every entity exceeds the lifetime of the Lua instance, you can use *static_cast<Entity**>(lua_newuserdata(state, sizeof(Entity*))) = self;, i.e. allocate a pointer-sized userdata and store the entity pointer there. No need for anything complex here.

If you manage the entity through a smart pointer, you can allocate it in the Lua instance and take advantage of its GC. In this case, the code is a bit more complex however:

#include <type_traits>

#if __GNUG__ && __GNUC__ < 5
#define is_trivially_constructible(T) __has_trivial_constructor(T)
#define is_trivially_destructible(T) __has_trivial_destructor(T)
#else
#define is_trivially_constructible(T) std::is_trivially_constructible<T>::value
#define is_trivially_destructible(T) std::is_trivially_destructible<T>::value
#endif

namespace lua
{
  template <class Type>
  struct _udata
  {
      typedef Type &type;

      static Type &to(lua_State *L, int idx)
      {
          return *reinterpret_cast<Type*>(lua_touserdata(L, idx));
      }

      static Type &check(lua_State *L, int idx, const char *tname)
      {
          return *reinterpret_cast<Type*>(luaL_checkudata(L, idx, tname));
      }
  };

  template <class Type>
  struct _udata<Type[]>
  {
      typedef Type *type;

      static Type *to(lua_State *L, int idx)
      {
          return reinterpret_cast<Type*>(lua_touserdata(L, idx));
      }

      static Type *check(lua_State *L, int idx, const char *tname)
      {
          return reinterpret_cast<Type*>(luaL_checkudata(L, idx, tname));
      }
  };

  template <class Type>
  typename _udata<Type>::type touserdata(lua_State *L, int idx)
  {
      return _udata<Type>::to(L, idx);
  }

  template <class Type>
  typename _udata<Type>::type checkudata(lua_State *L, int idx, const char *tname)
  {
      return _udata<Type>::check(L, idx, tname);
  }

  template <class Type>
  Type &newuserdata(lua_State *L)
  {
      auto data = reinterpret_cast<Type*>(lua_newuserdata(L, sizeof(Type)));
      if(!is_trivially_constructible(Type))
      {
          new (data) Type();
      }
      if(!is_trivially_destructible(Type))
      {
          lua_createtable(L, 0, 2);
          lua_pushboolean(L, false);
          lua_setfield(L, -2, "__metatable");
          lua_pushcfunction(L, [](lua_State *L) {
              lua::touserdata<Type>(L, -1).~Type();
              return 0;
          });
          lua_setfield(L, -2, "__gc");
          lua_setmetatable(L, -2);
      }
      return *data;
  }

  template <class Type>
  void pushuserdata(lua_State *L, const Type &val)
  {
      newuserdata<Type>(L) = val;
  }

  template <class Type, class=typename std::enable_if<!std::is_lvalue_reference<Type>::value>::type>
  void pushuserdata(lua_State *L, Type &&val)
  {
      newuserdata<typename std::remove_reference<Type>::type>(L) = std::move(val);
  }
}

If self is std::shared_ptr<Entity> you can then do lua::pushuserdata(state, self); and it will take care of everything ‒ properly initialize the userdata, and add a __gc metamethod that frees it. You may also do lua::pushuserdata<std::weak_ptr<Entity>>(state, self); if you don't want to let Lua prolong the lifetime of the entities. In-place construction is also possible too, if you modify lua::newuserdata accordingly.

Thank you so much for your response! So basically is amost always better to create Entities in Lua instead? I will have to rethink my approach. But this might be problematic if I want to have multiple entities defined in different scripts and have them interact with each other, do I have to use one state for all scripts or multiple states? — kaktusas2598, Dec 11 '22 at 12:57
If you can do that, it's the best way, but it's not necessary. It depends on how your EntityManager works, for example you could store some entity ID in the userdata instead of the pointer, and have the entity manager determine if the entity exists or not. It all boils down to how you can guarantee that an invalid entity will not be accessed ‒ if you don't have to, just storing the pointer is fine, but if not, you have to think of something that works the best with your code. — IS4, Dec 11 '22 at 13:06
So everytime in Lua I have test = Entity:create() which in C++ calls: EntityManager::addEntity(Entity * entity), but I think this is causing memory leaks, I am adding entity pointer to std::vector of unique_ptr, so I am thinking that if I replace this to std::map? Key would be unique ID assigned to each entity and value will be std::shared_ptr to Entity. When everytime lua calls _gc it will remove that Entity from the map using id. Also this why all the entities will be created in Lua and my callbacks should work, right? — kaktusas2598, Dec 11 '22 at 13:41
I have gotten it to work! Thank you! Basically I implemented another Id Component for each entity which is just a static counter for each Entity. Now I create all entities in Lua and have a global table mapped against Id, on C++ side I kept EntityManager which holds vector of unique pointers or should they be shared? Every time lua creates an entity, I create a pointer and move it to EntityManager vector, if ENtity is destroyed it gets removed from vector, seems to work. I am passing Ids instead of Entity pointers to Lua — kaktusas2598, Dec 11 '22 at 17:39
@kaktusas2598 Yeah that's fine I think. No need for shared_ptr if you don't have to use it. — IS4, Dec 11 '22 at 19:25

Not able to modify C++ object passed as parameter to Lua function

Working parts

Problem

1 Answers1