How to reinitialise an embedded Python interpreter?

Question

I'm working on embedding Python in our test suite application. The purpose is to use Python to run several tests scripts to collect data and make a report of tests. Multiple test scripts for one test run can create global variables and functions that can be used in the next script.

The application also provides extension modules that are imported in the embedded interpreter, and are used to exchange some data with the application.

But the user can also make multiple test runs. I don't want to share those globals, imports and the exchanged data between multiple test runs. I have to be sure I restart in a genuine state to control the test environment and get the same results.

How should I reinitialise the interpreter?

I used Py_Initialize() and Py_Finalize(), but get an exception on the second run when initialising a second time the extension modules I provide to the interpreter. And the documentation warns against using it more than once.

Using sub-interpreters seems to have the same caveats with extension modules initialization.

I suspect that I'm doing something wrong with the initialisation of my extension modules, but I fear that the same problem happens with 3rd party extension modules.

Maybe it's possible to get it to work by launching the interpreter in it's own process, so as to be sure that all the memory is released.

By the way, I'm using boost-python for it, that also warns AGAINST using Py_Finalize!

Any suggestion?

Thanks

score 6 · Accepted Answer · answered Apr 03 '12 at 11:44

Here is another way I found to achieve what I want, start with a clean slate in the interpreter.

I can control the global and local namespaces I use to execute the code:

// get the dictionary from the main module
// Get pointer to main module of python script
object main_module = import("__main__");
// Get dictionary of main module (contains all variables and stuff)
object main_namespace = main_module.attr("__dict__");

// define the dictionaries to use in the interpreter
dict global_namespace;
dict local_namespace;

// add the builtins
global_namespace["__builtins__"] = main_namespace["__builtins__"];

I can then use use the namespaces for execution of code contained in pyCode:

exec( pyCode, global_namespace, lobaca_namespace );

I can clean the namespaces when I want to run a new instance of my test, by cleaning the dictionaries:

// empty the interpreters namespaces
global_namespace.clear();
local_namespace.clear();        

// Copy builtins to new global namespace
global_namespace["__builtins__"] = main_namespace["__builtins__"];

Depending at what level I want the execution, I can use global = local

Aleksey Vitebskiy · Answer 2 · 2011-10-14T01:25:56.030

How about using code.IteractiveInterpreter?

Something like this should do it:

#include <boost/python.hpp>
#include <string>
#include <stdexcept>

using namespace boost::python;

std::string GetPythonError()
{
    PyObject *ptype = NULL, *pvalue = NULL, *ptraceback = NULL;
    PyErr_Fetch(&ptype, &pvalue, &ptraceback);
    std::string message("");
    if(pvalue && PyString_Check(pvalue)) {
        message = PyString_AsString(pvalue);
    }
    return message;
}

// Must be called after Py_Initialize()
void RunInterpreter(std::string codeToRun)
{
    object pymodule = object(handle<>(borrowed(PyImport_AddModule("__main__"))));
    object pynamespace = pymodule.attr("__dict__");

    try {
        // Initialize the embedded interpreter
        object result = exec(   "import code\n"
                                "__myInterpreter = code.InteractiveConsole() \n", 
                                pynamespace);
        // Run the code
        str pyCode(codeToRun.c_str());
        pynamespace["__myCommand"] = pyCode;
        result = eval("__myInterpreter.push(__myCommand)", pynamespace);
    } catch(error_already_set) {
        throw std::runtime_error(GetPythonError().c_str());
    }
}

So basically, I should instantiate once my Python interpreter, and use this interpreter to launch sub-interpreters, that all have their very own namespace ? Looks like a viable solution, if these subinterpreters don't suffer from the same caveats as the ones made with Py_NewInterpreter. I shall look in details into that and experiment with it. Thanks! — nab, Oct 14 '11 at 14:59
You got it. Instanciating an InteractiveInterpreter gives you a fresh environment every time. I'm not sure what the rules are regarding what's inherited from the parent interpreter, but that should be easy to control either way. — Aleksey Vitebskiy, Oct 15 '11 at 13:33
Seems to be doing what I want. Note that it has the same shortcoming for modules initialisation (they are only initialised once). But it works well to clean up the namespace. Thanks! — nab, Nov 02 '11 at 07:58

score 0 · Answer 3 · answered Oct 14 '11 at 07:10

I'd write another shell script executing the sequence of test scripts with new instances of python each time. Or write it in python like

# run your tests in the process first
# now run the user scripts, each in new process to have virgin env
for script in userScript:
    subprocess.call(['python',script])

How to reinitialise an embedded Python interpreter?

3 Answers3

Linked