Python runtime: recompiling and reusing C library

Question

I am developing a tool for some numerical analysis of user-defined functions. The idea is to make a convenient UI in Python, where user can enter C function, then press a button - and receive some output data. Computations can take minutes or hours, so Numpy-only performance is not acceptable.

I have tried the following approach: the Python-based UI calls gcc, compiles dll from user functions that is than used by my core C-based algorithms in Cython wrappings. It works, but since there is no way to fully unload the python module, I can not recompile user-defined function until the whole UI program is closed and run again.

The only way I see now is to separate the computational core and UI processes and then make them interact via shared memory/messaging. As user wants to update his function, the program terminates the core, recompiles dll and starts the core again.

Can you suggest any common practice in such cases?

Thank you!

Answer 1

Python has really good multiprocessing support (and really not very good threading support), so you could spawn a new python process for each expression to be evaluated, compile and load the dll in the new process, then have it communicate the results back to the parent process. When the spawned process exits, everything should be unloaded.

Answer 2

Each time the user enters a function, you could generate a new module with a random, unique name (as you might do for a temp file). Then compile that module, load it, and call the function.

Over time, this will of course result in a lot of modules being loaded. If users don't run too many functions in a single session, you might get away with it. Alternatively, you could compile the function within a plain DLL instead of a Python module, and load it with ctypes. When you're done with the DLL, simply discard any references to it -- when the ctypes handle is eventually gced, it should unload the DLL (see How can I unload a DLL using ctypes in Python?).

You will also probably want to delete your DLLs from disk. If you're using Windows, that might be tricky, since it's hard to predict when ctypes will unload it, and Windows won't let you delete it will it's in use.

Answer 3

Have you considered Weave or Instant (less up-to-date)? I've only used instant myself, but it seems like either would be perfect for what you're describing.

They can both automatically compile and cache c code at runtime from a string. I've only ever used instant to inline c, but I'm sure it or weave would work just as well for compiling user input.

Here's an example of how to use weave, from the tutorial.

>>> a = 'string'
>>> def protected_printf(a):
...     assert(type(a) == type(1))
...     weave.inline(r'printf("%d\n",a);',['a'])
>>> protected_printf(1)
 1
>>> protected_printf('string')
AssertError...