Python 3 C-API IO and File Execution

Question

I am having some serious trouble getting a Python 2 based C++ engine to work in Python3. I know the whole IO stack has changed, but everything I seem to try just ends up in failure. Below is the pre-code (Python2) and post code (Python3). I am hoping someone can help me figure out what I'm doing wrong.I am also using boost::python to control the references.

The program is supposed to load a Python Object into memory via a map and then upon using the run function it then finds the file loaded in memory and runs it. I based my code off an example from the delta3d python manager, where they load in a file and run it immediately. I have not seen anything equivalent in Python3.

Python2 Code Begins here:

    // what this does is first calls the Python C-API to load the file, then pass the returned
    // PyObject* into handle, which takes reference and sets it as a boost::python::object.
    // this takes care of all future referencing and dereferencing.
    try{
        bp::object file_object(bp::handle<>(PyFile_FromString(fullPath(filename), "r" )));
        loaded_files_.insert(std::make_pair(std::string(fullPath(filename)), file_object));
    }
    catch(...)
    {
        getExceptionFromPy();
    }

Next I load the file from the std::map and attempt to execute it:

    bp::object loaded_file = getLoadedFile(filename);
    try
    {
        PyRun_SimpleFile( PyFile_AsFile( loaded_file.ptr()), fullPath(filename) );
    }
    catch(...)
    {
        getExceptionFromPy();
    }

Python3 Code Begins here: This is what I have so far based off some suggestions here... SO Question Load:

        PyObject *ioMod, *opened_file, *fd_obj;

        ioMod = PyImport_ImportModule("io");
        opened_file = PyObject_CallMethod(ioMod, "open", "ss", fullPath(filename), "r");

        bp::handle<> h_open(opened_file);
        bp::object file_obj(h_open);
        loaded_files_.insert(std::make_pair(std::string(fullPath(filename)), file_obj));

Run:

    bp::object loaded_file = getLoadedFile(filename);
    int fd = PyObject_AsFileDescriptor(loaded_file.ptr());
    PyObject* fileObj = PyFile_FromFd(fd,fullPath(filename),"r",-1,"", "\n","", 0);

    FILE* f_open = _fdopen(fd,"r");

    PyRun_SimpleFile( f_open, fullPath(filename) );

Lastly, the general state of the program at this point is the file gets loaded in as TextIOWrapper and in the Run: section the fd that is returned is always 3 and for some reason _fdopen can never open the FILE which means I can't do something like PyRun_SimpleFile. The error itself is a debug ASSERTION on _fdopen. Is there a better way to do all this I really appreciate any help.

If you want to see the full program of the Python2 version it's on Github

I'm confused at to what you are trying to do. Which of the above Python 3 examples are supposed to replace which of the above Python 2 examples? They are all so different. And what is your actual error message? — Lennart Regebro, Jan 05 '11 at 15:37
Fair enough, I made some edits and explained in the second paragraph a little more on what I'm trying to accomplish. — Kyle C, Jan 05 '11 at 15:42
I may have found the solution, but I can't test it until I get home. When I do I will post it. — Kyle C, Jan 05 '11 at 20:35
Yeah, sorry, that didn't clarify it at all. There are four snippets of code, and I don't understand how they relate to each other. — Lennart Regebro, Jan 05 '11 at 22:08

Kyle C · Accepted Answer · 2011-02-08T19:01:41.913

So this question was pretty hard to understand and I'm sorry, but I found out my old code wasn't quite working as I expected. Here's what I wanted the code to do. Load the python file into memory, store it into a map and then at a later date execute that code in memory. I accomplished this a bit differently than I expected, but it makes a lot of sense now.

Open the file using ifstream, see the code below
Convert the char into a boost::python::str
Execute the boost::python::str with boost::python::exec
Profit ???

Step 1)

vector<char> input;
ifstream file(fullPath(filename), ios::in);
if (!file.is_open())
{
    // set our error message here
    setCantFindFileError();
    input.push_back('\0');
    return input;
}

file >> std::noskipws;
copy(istream_iterator<char>(file), istream_iterator<char>(), back_inserter(input));
input.push_back('\n');
input.push_back('\0');

Step 2) bp::str file_str(string(&input[0])); loaded_files_.insert(std::make_pair(std::string(fullPath(filename)), file_str)); Step 3)

bp::str loaded_file = getLoadedFile(filename);
// Retrieve the main module
bp::object main = bp::import("__main__");
// Retrieve the main module's namespace
bp::object global(main.attr("__dict__"));
bp::exec(loaded_file, global, global);

Full Code is located on github:

Python 3 C-API IO and File Execution

1 Answers1