4

Sometimes, I want to use lua_pushstring in places after I allocated some resources which I would need to cleanup in case of failure. However, as the documentation seems to imply, lua_push* functions can always end up with an out of memory exception. But that exception instant-quits my C scope and doesn't allow me to cleanup whatever I might have temporarily allocated that might have to be freed in case of error.

Example code to illustrate the situation:

void* blubb = malloc(20);
...some other things happening here...
lua_pushstring(L, "test"); //how to do this call safely so I can still take care of blubb?
...possibly more things going on here...
free(blubb);

Is there a way I can check beforehand if such an exception would happen and then avoid pushing and doing my own error triggering as soon as I safely cleaned up my own resources? Or can I somehow simply deactivate the setjmp, and then check some "magic variable" after doing the push to see if it actually worked or triggered an error?

I considered pcall'ing my own function, but even just pushing the function on the stack I want to call safely through pcall can possibly give me an out of memory, can't it?

To clear things up, I am specifically asking this for combined use with custom memory allocators that will prevent Lua from allocating too much memory, so assume this is not a case where the whole system has run out of memory.

E. T.
  • 847
  • 10
  • 26

3 Answers3

2

Unless you have registered a user-defined memory handler with Lua when you created your Lua state, getting an out of memory error means that your entire application has run out of memory. Recovery from this state is generally not possible. Or at least, not feasible in a lot of cases. It could be depending on your application, but probably not.

In short, if it ever comes up, you've got bigger things to be concerned about ;)

The only kind of cleanup that should affect you is for things external to your application. If you have some process global memory that you need to free or set some state in. You're doing interprocess communication and you have some memory mapped file you're talking though. Or something like that.

Otherwise, it's probably better to just kill your process.

You could build Lua as a C++ library. When you do that, errors become actual exceptions, which you can either catch or just use RAII objects to handle.

If you're stuck with C... well, there's not much you can do.

I am specifically interested in a custom allocator that will out of memory much earlier to avoid Lua eating too much memory.

Then you should handle it another way. To signal an out-of-memory error is basically to say, "I want Lua to terminate right now."

The way to stop Lua from eating memory is to periodically check the Lua state's memory, and garbage collect it if it's using too much. And if that doesn't free up enough memory, then you should terminate the Lua state manually, but only when it is safe to do so.

Nicol Bolas
  • 449,505
  • 63
  • 781
  • 982
  • I am specifically interested in a custom allocator that will out of memory much earlier to avoid Lua eating too much memory. However, I am reluctant to do that because of the problems that go along with the out of memory exception. – E. T. Apr 22 '12 at 13:19
  • 1
    "To signal an out-of-memory error is basically to say, "I want Lua to terminate right now."" I don't agree that that's true, but even if it is that doesn't mean that I don't need to clean up before it does so – david king Apr 20 '16 at 21:41
0

lua_atpanic() may be one solution for you, depending on the kind of cleanup you need to do. It will never throw an error.

In your specific example you could also create blubb as a userdata. Then Lua would free it automatically when it left the stack.

MattJ
  • 7,924
  • 1
  • 28
  • 33
  • Use `lua_pcall` for local error recovery. `lua_atpanic` is not meant for error recovery at that level, only at the application level. Once a Lua state has reached panic status, it cannot be reused. – lhf Apr 22 '12 at 12:33
  • The problem with lua_pcall seems to be that I would need to push the pcalled function itself first, possibly leading to out of memory aswell. So I would need to pcall the pcall... or something :P it doesn't really solve the problem it seems – E. T. Apr 22 '12 at 13:22
0

I have recently gotten into some more Lua sandboxing again, and now I think the answer I accepted previously is a bad idea. I have given this some more thought:

Why periodic checking is not enough

Periodically checking for large memory consumption and terminating Lua "only when it is safe to do so" seems like a bad idea if you consider that a single huge table can eat up a lot of your memory with one single VM instruction about which you'll only find out after it happened - where your program might already be dying from it, and then you indeed have much bigger problems which you could have avoided entirely if you had stopped that allocation in time in the first place.

Since Lua has a nice out of memory exception already built-in, I would just like to use that one since this allows me to do the minimal required thing (preventing the script from allocating more stuff, while possibly allowing it to recover) without my C code breaking from it.

Therefore my current plan for Lua sandboxing with memory limit is:

  • Use custom allocator that returns NULL with limit

  • Design all C functions to be able to handle this without memory leak or other breakage

But how to design the C functions safely?

How to do that, since lua_pushstring and others can always setjmp away with an error without me knowing whether that is gonna happen in advance? (this was originally my question)

I think I found a working approach:

I added a facility to register pointers when I allocate them, and where I unregister them after I am done with them. This means if Lua suddenly setjmp's me out of my C code without me getting a chance to clean up, I have everything in a global list I need to clean up that mess later when I'm back in control.

Is that ugly or what?

Yes, it is quite the hack. But, it will most likely work, and unlike 'periodic checking' it will actually allow me to have a true hard limit and avoid getting the application itself trouble because of an aggressive attack.

E. T.
  • 847
  • 10
  • 26
  • In your example you should just use `lua_newuserdata` instead of `malloc`. The GC will eventually cleanup the temporary memory when the current stack frame is left. You can even build some form of RAII in Lua using this technique: Create a pointer sized userdata. Set it to NULL. Create a metatable with a `__gc` method that does your cleanup iff the userdata value is not NULL. Associate the metatable with the userdata. Allocate your resource and (if successful) store the pointer in the userdata. If at any point during this sequence (or later) an error is thrown, the GC can clean everything up. – siffiejoe Sep 08 '14 at 02:36
  • I'm not sure this is entirely practical in all situations - just because you manage something dynamically in a lua call doesn't mean you want to hand it out or have it owned by lua. It could be that you need to use your own allocating pool or do something else (not malloc) that needs to be cleaned up - in which case you will need to use the sort of hack I described. Your suggestion is nice *WHEN* it is applicable of course. Anyway, preferrably I think Lua should fix this and allow disabling setjmp temporarily. See also libpng as example (uses setjmp but optional, offers als oreturn values). – E. T. Aug 04 '15 at 11:54
  • 1
    Using the `__gc` metamethod as I described you can clean up any resource you want, even if it's allocated via `malloc` or some other function. So it's pretty much universally applicable, the only drawback being that if a `setjmp` happens, the GC may take some time before the resource is actually reclaimed. Anyway, I think it's pretty unlikely that Lua will add an option to disable `setjmp`. It would radically change the C API. – siffiejoe Aug 05 '15 at 00:15
  • @siffiejoe I thought about your suggestion, and lua_newuserdata (as well as adding the metatable) can itself cause an out of memory exception again, making this solution potentially quite complicated as well. I have come to the conclusion that maybe Lua simply isn't suitable for side-effect free sandboxing and I am looking for other options alternative to Lua at this point. – E. T. Sep 11 '17 at 03:42
  • Good luck to you! Apart from C and C++, Lua is the only language I feel comfortable handling out-of-memory errors in. For instance, Lua is [systematically tested in out-of-memory situations](http://lua-users.org/lists/lua-l/2008-10/msg00379.html). Some "system-programming" languages like e.g. Rust just die when memory runs out (which with the wrong kernel settings could happen to you anyways in any language). If you need memory to do a certain task and you don't have any, you **will** fail. The question is whether you remain in control while you fail. And I think in Lua you can do that. – siffiejoe Sep 11 '17 at 18:48
  • What do you mean with "you think"? We've already established that while in Lua you can, it does create huge trouble handling this nicely in C. That's, after all, why we're discussing this so much. (And I think that's legitimately a big shortcoming of Lua's setjmp that can't be turned off.) – E. T. Mar 13 '18 at 05:06
  • With "I think" I mean I have yet to come across a situation where the problem couldn't be solved by the following approach: Allocate a userdata with a `__gc` metamethod. The userdata is inactive, so the `__gc` won't do anything yet. Allocate your resource. "Activate" the userdata by setting a pointer to non-`NULL` or an integer to non-zero in the userdata memory. The userata stuff is generic and reusable, so you can put it into a library function. – siffiejoe Mar 17 '18 at 07:55
  • Allocating userdata isn't useful as a generic solution, it can throw an error itself before you're done setting it up with an attached resource (and no, using a full userdata is not always an option) which again will leak. It's not as easy as most people here make it sound - there are ways, but there are many nasty corner cases due to lua's setjmp silliness. I've come to terms with it that you either need to do external tracking with register/unregister of resources as described above to really cover all cases, or simply use another language with a better API. – E. T. Mar 17 '18 at 08:57
  • No problem. Creating the userdata may fail, creating the metatable may fail, but both are only chunks memory that Lua's GC knows about, so they will be cleaned up eventually. And you haven't allocated your external resource yet. Once you do, you only need to set one pointer/integer for Lua's GC to take over responsibility. – siffiejoe Mar 17 '18 at 11:33
  • Yes, if you couldn't prepare before being passed a resource, then you would indeed have a problem. That's why I said "I think". I haven't encountered such a case yet, and I can't see why someone would do it this way. So for me the simple approach I outlined above is sufficient to handle Lua's `setjmp` craziness, and as a result the Lua API can stay nice and clean. – siffiejoe Mar 17 '18 at 12:09