Tracing code execution in embedded Python interpreter

Question

I'd like to create an application with embedded python interpreter and basic debugging capabilities. Now I'm searching the API for functions which I could use to run code step-by-step and get the number of the current line of code which is being (or is about to be) executed.

Official Python docs seem a little underdone for me when comes it to tracing and profiling. There is, for example, no information about the meaning of the return value of Py_tracefunc.

So far I've assembled the following:

#include <Python.h>

static int lineCounter = 0;

int trace(PyObject *obj, PyFrameObject *frame, int what, PyObject *arg)
{
    if(what == PyTrace_LINE)
    {
        lineCounter += 1;
        printf("line %d\n", lineCounter);
    }
    return 0;
}

int main(int argc, char *argv[])
{
    wchar_t *program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL) {
        fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
        exit(1);
    }
    Py_SetProgramName(program);  /* optional but recommended */
    Py_Initialize();
    PyEval_SetTrace(trace, NULL);
    char *code = "def adder(a, b):\n"
                 " return a + b\n"
                 "x = 3\n"
                 "y = 4\n"
                 "print(adder(x, y))\n";
    PyRun_SimpleString(code);
    Py_Finalize();
    PyMem_RawFree(program);
    return 0;
}

However, the compiler outputs the following error:

hello.c:5:26: error: unknown type name ‘PyFrameObject’
 int trace(PyObject *obj, PyFrameObject *frame, int what, PyObject *arg)
                          ^

I'm operating on ManjaroLinux and using the following to compile the above:

gcc -o hello hello.c -I/usr/include/python3.5m  -Wno-unused-result -Wsign-compare -Wunreachable-code -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -L/usr/lib -lpython3.5m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic

I've found that I can replace PyFrameObject with struct _frame and then program compiles but everyone knows it's a dirty hack, not a solution.

The executable outputs the following:

line 1
line 2
line 3
line 4
line 5
7

But I'd like the traces to follow the execution flow of the script (that is: start from line 3, then 4, 5 and then, due to the function call, 2).

I could not find anything about step-by-step execution.

Could you recommend some other sources about Python C API with more information and some introduction to the topic?

I awarded the answer with bounty since it would expire anyway. However, I'm still looking and would be grateful for answers for other questions from above.

This might help regarding the trace function: https://docs.python.org/3/library/sys.html#sys.settrace — Sven Marnach, Dec 03 '15 at 20:56
All of the CPython source code, actually, not only the header files. You are working with implementation details of the CPython interpreter here, and in these cases, it's often necessary (o at least helpful) to look at the source code in addition to the documentation. E.g. see [the definition of `Py_tracefunc`](https://github.com/python/cpython/blob/1fe0fd9feb6a4472a9a1b186502eb9c0b2366326/Include/pystate.h#L53) for the meaning of the return value. — Sven Marnach, Dec 04 '15 at 11:26
@SvenMarnach: "return -1 when raising an exception, or 0 for success" - does it mean that I should return -1 when my implementation of that callback raises an exception? Why would I raise an exception there? — Luke, Dec 04 '15 at 11:54
Well, read the source code. Here's where the trace function is called: https://github.com/python/cpython/blob/aed79b41a1fbcedd4697269e3fdd40af5ee82b14/Python/ceval.c#L4352. You can look for the callers of this function (in the same file), and see how it behaves if you return an error (it won't enter the frame). If you don't want this to happen, you probably don't want to throw an exception. — Sven Marnach, Dec 04 '15 at 12:08

score 9 · Answer 1 · edited Dec 09 '15 at 22:12

9

hello.c:5:26: error: unknown type name ‘PyFrameObject’

This error means that PyFrameObject has not been declared. I did a Google search which showed me frameobject.h in the Python source tree is where that structure is declared.

I expect that you can add the line

#include <frameobject.h>

to resolve this.

edited Dec 09 '15 at 22:12

Iharob Al Asimi

52,653
6
59
97

answered Dec 03 '15 at 21:12

dsh

12,037
3
33
51

I read somewhere on the Internet that Python.h alone should be sufficient. The example from the docs also includes only Python.h. – Luke Dec 03 '15 at 21:40
1

Could you provide any source claiming that frameobject.h should be included separately? – Luke Dec 04 '15 at 11:43
I inferred it from the error message you posted. You can determine for yourself by reading the source. Read Python.h and follow the various includes to see where/if it includes frameobject.h. It might even be conditional based on defined preprocessor tokens. – dsh Dec 07 '15 at 21:11
For that much I could just trust you. Too bad the docs are pretty selective. But that simple compilation error isn't really my biggest worry, as you can probably guess. – Luke Dec 08 '15 at 07:07
@Luke `Python.h` should be sufficient. However, it could be a bug in python source code – Xavier Combelle Dec 10 '15 at 15:35
Some other python headers contain the forward declaration of `struct _frame` "to avoid including frameobject.h". You can substitute `PyFrameObject` with `struct _frame`. One could search for the cause in the git history of python. – payload Dec 13 '15 at 18:43
I awarded your answer with bounty since it would expire anyway. However, I'm still looking and would be grateful for the answers for my other questions. – Luke Dec 14 '15 at 07:27

score 0 · Answer 2 · edited Aug 10 '17 at 20:05

The pyFrameObject has a

int f_lineno;

field. You can use it. But apparently, it is not always storing the correct value. So, you should probably use the function:

/* Return the line of code the frame is currently executing. */
int PyFrame_GetLineNumber(PyFrameObject *);

then, you can use

frame->f_code->co_filename

to get the current file name

frame->f_code->co_name

to get the current function name and

frame->f_back

to get one level down in the call stack. .

score 0 · Answer 3 · edited Nov 02 '22 at 04:31

0

According to docstring in Python v3.10.8 in Include/cpython/pystate.h:

/* Py_tracefunc return -1 when raising an exception, or 0 for success. */
typedef int (*Py_tracefunc)(PyObject *, PyFrameObject *, int, PyObject *);

edited Nov 02 '22 at 04:31

Community

1
1

answered Oct 27 '22 at 15:43

Tanner Firl

80
6

1

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Oct 31 '22 at 07:35

score -1 · Answer 4 · edited Oct 25 '16 at 19:16

-1

PyFrameObject is just a _frame struct. Just replace PyFrameObject by _frame in your function signature and you won't have to include any additional python headers.

edited Oct 25 '16 at 19:16

mnille

1,328
4
16
20

answered Oct 25 '16 at 17:01

John McKowen

1

Tracing code execution in embedded Python interpreter

4 Answers4