boost python, using a namespace other than main global

Question

I am embedding python in my C++ application using boost python. I am a C++ programmer, with very limited knowledge of Python.

I have a C++ class, PyExpression. Each instance of this class has a string expStr, which is a short user-entered (at runtime) python program, that is executed by calling boost::python::exec. Briefly, I have this set up as:

//import main and its globals
bp::object main = bp::import("__main__");
bp::object main_namespace = main.attr("__dict__");

where main and main_namespace are members of the C++ class PyExpression.

void PyExpression::Run()
{
    bp::object pyrun = exec(expStr,main_namespace);
}

The problem here is that different C++ instances of PyExpression modify the same global python namespace, main_namespace, and I want each PyExpression instance to have its own "global" namespace.

If I pass in boost::python::dict class_dict instead of main_namespace above, it works at a basic level. But if PyExpression::expStr imports a module, e.g. import sys, then I get an ImportError. Also, using class_dict, I can no longer call globals(), locals(), vars(), as they all become undefined.

I have also tried exposing PyExpression as a python module. Briefly,

BOOST_PYTHON_MODULE(PyExpModule)
{
    bp::class_<PyExpression>("PyExpression", bp::no_init)
    //a couple .def functions
}

int pyImport = PyImport_AppendInittab( "PyExpModule", &initPyExpModule );

bp::object thisExpModule = bp::object( (bp::handle<>(PyImport_ImportModule("PyExpModule"))) );
bp::object PyExp_namespace = thisExpModule.attr("__dict__");

Unfortunately, using PyExp_namespace, again I get the ImportError when the string to be executed imports a python module, and again, the namespace is shared between all instances of PyExpression.

In short, I want to be able to use a namespace object/dictionary, that is preferably a class member of PyExpression, have only that instance of PyExpression have access to the namespace, and the namespace to act like a global namespace such that other modules can be imported, and the `globals(), locals(), vars() are all defined.

If anyone can point me to a sketch of working code, I would very much appreciate it. I can't find relevant material on this problem.

score 3 · Accepted Answer · answered May 16 '13 at 18:40

Before providing a solution, I want to provide some clarification on Python behavior.

Boost.Python's object is essentially a higher-level handle of a smart pointer. Thus, multiple object instances may point to the same Python object.

object main_module = import("__main__");
object main_namespace = main_module.attr("__dict__");

The above code imports a module named __main__. In Python, modules are essentially singletons due to the import behavior. Therefore, although the C++ main_module may be a member of the C++ PyExpression class, they all point to the same Python __main__ module, as it is a singleton. This results in main_namespace pointing to the same namespace.

Much of Python is built around dictionaries. For example, with an example module:

class Foo:
    def __init__(self):
        self.x = 42;

    def bar(self):
        pass

There are 3 dictionaries of interests:

example.__dict__ is the example module's namespace.

>>> example.__dict__.keys()
['__builtins__', '__file__', '__package__', '__name__', 'Foo', '__doc__']

example.Foo.__dict__ is a dictionary that describes the Foo class. Additionally, it would contain the equivalent of C++'s static member variables and functions.
```
>>> example.Foo.__dict__.keys()
['__module__', 'bar', '__doc__', '__init__']
```
example.Foo().__dict__ is a dictionary containing instance-specific variables. This would contain the equivalent of C++'s non-static member variables.
```
>>> example.Foo().__dict__.keys()
['x']
```

The Python exec statement takes two optional arguments:

The first argument specifies the dictionary that will be used for globals(). If the second argument is omitted, then it is also used for locals().
The second argument specifies the dictionary that will be used for locals(). Variable changes occurring within exec are applied to locals().

To get the desired behavior, example.Foo().__dict__ needs to be used as locals(). Unfortunately, this becomes slightly more complicated because of the following two factors:

Although import is a Python keyword, the CPython implementation is dependent on __builtins__.__import__. Thus, there needs to be a guarantee that the __builtin__ module is assessable as __builtins__ within the namespace passed to exec.
If a C++ class called Foo is exposed as a Python class through Boost.Python, then there is no easy way to access the Python Foo instance from within the C++ Foo instance.

To account for these behaviors, the C++ code will need to:

Get a handle to the Python object's __dict__.
Inject the __builtin__ module into the Python object's __dict__.
Extract the C++ object from the Python object.
Pass the Python object's __dict__ to the C++ object.

Here is an example solution that only sets variables on the instance for which code is being evaluated:

#include <boost/python.hpp>

class PyExpression
{
public:
  void run(boost::python::object dict) const
  {
    exec(exp_.c_str(), dict);
  }
  std::string exp_;
};

void PyExpression_run(boost::python::object self)
{
  // Get a handle to the Python object's __dict__.
  namespace python = boost::python;
  python::object self_dict = self.attr("__dict__");

  // Inject the __builtin__ module into the Python object's __dict__.
  self_dict["__builtins__"] = python::import("__builtin__");

  // Extract the C++ object from the Python object.
  PyExpression& py_expression = boost::python::extract<PyExpression&>(self);

  // Pass the Python object's `__dict__` to the C++ object.
  py_expression.run(self_dict);
}

BOOST_PYTHON_MODULE(PyExpModule)
{
  namespace python = boost::python;
  python::class_<PyExpression>("PyExpression")
    .def("run", &PyExpression_run)
    .add_property("exp", &PyExpression::exp_, &PyExpression::exp_)
    ;
}

// Helper function to check if an object has an attribute.
bool hasattr(const boost::python::object& obj,
             const std::string& name)
{
  return PyObject_HasAttrString(obj.ptr(), name.c_str());
}

int main()
{
  PyImport_AppendInittab("PyExpModule", &initPyExpModule);
  Py_Initialize();

  namespace python = boost::python;
  try
  {
    // python: import PyExpModule
    python::object py_exp_module = python::import("PyExpModule");

    // python: exp1 = PyExpModule.PyExpression()
    // python: exp1.exp = "import time; x = time.localtime().tm_year"
    python::object exp1 = py_exp_module.attr("PyExpression")();
    exp1.attr("exp") = 
      "import time;"
      "x = time.localtime().tm_year"
      ;

    // python: exp2 = PyExpModule.PyExpression()
    // python: exp2.exp = "import time; x = time.localtime().tm_mon"
    python::object exp2 = py_exp_module.attr("PyExpression")();
    exp2.attr("exp") = 
      "import time;"
      "x = time.localtime().tm_mon"
      ;

    // Verify neither exp1 nor exp2 has an x variable.
    assert(!hasattr(exp1, "x"));
    assert(!hasattr(exp2, "x"));

    // python: exp1.run()
    // python: exp2.run()
    exp1.attr("run")();
    exp2.attr("run")();

    // Verify exp1 and exp2 contain an x variable.
    assert(hasattr(exp1, "x"));
    assert(hasattr(exp2, "x"));

    // python: print exp1.x
    // python: print exp2.x
    std::cout << python::extract<int>(exp1.attr("x")) 
      << "\n" << python::extract<int>(exp2.attr("x"))
      << std::endl;
  }
  catch (python::error_already_set&)
  {
    PyErr_Print();
  }
}

And the output:

[twsansbury@localhost]$ ./a.out 
2013
5

Due to how libraries are loaded from imports, it may require providing arguments to the linker that will cause all symbols, not only used ones, to the dynamic symbol table. For example, when compiling the above example with gcc, using -rdynamic was required. Otherwise, import time will fail due to an undefined PyExc_IOError symbol.

score 0 · Answer 2 · edited May 23 '17 at 12:05

Python does not provide a 100% reliable isolation mechanism for this kind of task. That said, the essential tool you are looking for is the Python C-API Py_NewInterpreter, which is documented here. You will have to call it upon the creation of your PyExpression object, to create a new (semi)-isolated environment (N.B.: the destructor should call Py_EndInterpreter).

This is untested, but I'd guess something liket this would do the job:

PyThreadState* current_interpreter = Py_NewInterpreter();
bp::object pyrun = exec(expStr);
Py_EndInterpreter(current_interpreter);

You may wrap that into an object. If you wish to do so, you must manage the "thread" state as explained in this other stackoverflow thread.

boost python, using a namespace other than main global

2 Answers2

Linked