How to annotate lines with code when running PyRun_String() in Python C API?

Question

I'm using PyRun_String() from Python C API to run python code.

Passing Py_file_input for start and for globals and locals I pass dictionary created with PyDict_New(), and for string of code str I pass my code.

For example I'm having next code:

def f():
    def g():
        assert False, 'TestExc'
    g()
f()

Of cause this code is expected to throw an exception and show stack. I print error with PyErr_Print() and get next stack:

Traceback (most recent call last):
  File "<string>", line 5, in <module>
  File "<string>", line 4, in f
  File "<string>", line 3, in g
AssertionError: TestExc

As one can see this exception stack is missing lines with code, for example if same script is run in pure Python interpreter then it prints next stack:

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    f()
  File "test.py", line 4, in f
    g()
  File "test.py", line 3, in g
    assert False, 'TestExc'
AssertionError: TestExc

So it has code annotation e.g. assert False, 'TestExc' for last line of stack and f() and g() for previous lines. Also it has file name (but having file name is not very important).

Is there any way to have this code shown when using PyRun_String()? I expect that I need to use another function for example PyRun_StringFlags() which has extra parameters PyCompilerFlags * flags that I can probably use to tell compiler to save code attached to each line when compiling. But I don't see anywhere documentation for PyCompilerFlags and don't know what should I pass.

Also maybe there are other useful flags for PyCompilerFlags, for example would be nice to have test.py file name instead of <string> inside exception stack, probably this behaviour is also tweakable by some of PyCompilerFlags values?

Also I was using exec() from built-ins of Python, passing string with program's code to it. But got same exception stack without code annotation. Seems that if it is some interpreter-wide param whether to save code annotation or not.

I also tried to write special function to get current stack, using standard traceback and sys modules:

def GetStack():
    import traceback, sys
    frame = sys._getframe()
    extracted = traceback.extract_stack(frame)
    def AllFrames(f):
        while f is not None:
            yield f
            f = f.f_back
    all_frames = list(reversed(list(AllFrames(frame))))
    assert len(extracted) == len(all_frames)
    return [{
        'file': fs.filename, 'first_line': fr.f_code.co_firstlineno,
        'line': fs.lineno, 'func': fs.name, 'code': fs._line,
    } for fs, fr in zip(extracted, all_frames)]

This function returns whole stack correctly, ut inside code fields there are empty strings. Looks like frame objects don't have code annotation inside theirs ._line attribute as they probably should, this might be the reason of having no code annotation inside all the functions used above.

Do you know if there is any way to provide code annotation for all stack retrieving/printing operations? Besides manually composing correct stack trace by hand. Maybe there is at least some standard module that allows to set this lines of code somehow at least manually?

Update. I found out that traceback module uses linecache.getline(self.filename, self.lineno) (see here) to get source code. Does anybody know how can I fill linecache with source text from memory with given filename without using temporary file?

Also interesting if raised exception uses traceback module to output exception to console or maybe it has its own formatting implementation?

Answer 1

Answering my own question. After reading source code of PyRun_String() I found out that it is impossible to annotate code-lines of exception (unless I missed something).

Because PyRun_String() sets filename to "<string>" and doesn't allow to give other name, while exception printing code tries to read file from file system, and of course doesn't find this file name.

But I found out how to use Py_CompileString() with PyEval_EvalCode() to achieve line annotation instead of using just PyRun_String().

Basically I create temporary file with the help of tempfile standard module. You can create non-temporary file too, doesn't matter. Then write source code to this file and provide file name to Py_CompileString(). After this, lines are annotated correctly.

Below code is in C++, although you can use it in C too with small tweaks (like using PyObject * instead of auto).

Important. For the sake of simplicity in my code I don't handle errors of all function, also don't do reference counting of objects. Also finally I don't delete temporary file. These all things should be done in real programs. Hence code below can't be used in production directly, without modifications.

Try it online!

#include <Python.h>

int main() {
    Py_SetProgramName(L"prog.py");
    Py_Initialize();
    
    char const source[] = R"(
def f():
    def g():
        assert False, 'TestExc'
    g()
f()
)";
    
    auto pTempFile = PyImport_ImportModule("tempfile");
    
    auto pDeleteFalse = PyDict_New();
    PyDict_SetItemString(pDeleteFalse, "delete", Py_False);
    
    auto pFile = PyObject_Call(
        PyObject_GetAttrString(pTempFile, "NamedTemporaryFile"),
        PyTuple_Pack(0), pDeleteFalse);
    auto pFileName = PyObject_GetAttrString(pFile, "name");
    
    PyObject_CallMethod(pFile, "write", "y", source);
    PyObject_CallMethod(pFile, "close", nullptr);
    
    auto pCompiled = Py_CompileString(
        source, PyUnicode_AsUTF8(pFileName), Py_file_input);
    auto pGlobal = PyDict_New(), pLocal = PyDict_New();
    auto pEval = PyEval_EvalCode(pCompiled, pGlobal, pLocal);
    PyErr_Print();
    
    Py_FinalizeEx();
}

Output:

Traceback (most recent call last):
  File "C:\Users\User\AppData\Local\Temp\tmp_73evamv", line 6, in <module>
    f()
  File "C:\Users\User\AppData\Local\Temp\tmp_73evamv", line 5, in f
    g()
  File "C:\Users\User\AppData\Local\Temp\tmp_73evamv", line 4, in g
    assert False, 'TestExc'
AssertionError: TestExc