I've managed to get BoehmGC working on webassembly by forcing all roots on the "side stack". This is all working fine except for cases where there are lots of finalizers. It all works if I don't have finalizers, no matter how many objects I allocate. As soon as I set a finalizer it fails because an object is corrupted. The finalizer seems to get called on an invalid object (the value at 0 was invalid or it was never allocated in the first place).
Now I realize it sounds a bit vague, but does this ring a bell to anyone?
__GC_Finalizer
GC_notify_or_invoke_finalizers
GC_generic_malloc
GC_malloc
It alwyas fails on this callstack.
__GC_Finalizer gets called tons of times with success and does nothing more than call a vmt method. Pseudo code looks like:
(*(FnPtrWithThis**)obj))[8](obj)
It fails on this already, where obj does not point to a valid vmt (all which are constant in the data segment, but the vmt it points to isn't a real vmt address).
If anyone has some info I could look into, like docs on how the finalizers work internally, or places I could debug, really any hints would be appreciated.