How to access sys.stderr after calling PyErr_Print with python c-api

Question

I am trying to use the python c-api from a c# project via dll import.

I am getting an ModuleNotFoundError when importing some modules, which I thought are builtin. (Note: I compiled python myself)

I am a bit stuck right now, but my hope is to get some extra information when calling PyErr_Print() in the code below.

Code:

IntPtr modulePtr = NativeInterface.PyImport_ExecCodeModuleEx(moduleName,compiledModule, path);
if (modulePtr == IntPtr.Zero)
{
  NativeInterface.PyErr_Print();
  PythonException exception = PythonException.Query();
  throw exception;
}

The docs for PyErr_Print state that it will populate sys.stderr with some error information. What would be the easiest way to read this variable from my c# application?

I don't know enough C# to really make this an answer but i'd do something like: 1) replace `sys.stdout` with a `BytesIO` object. 2) Get the contents of that `BytesIO` object with `PyBytes_AsString`. Both steps are a little more involved than I've made out, but you can probably write that code in Python. — DavidW, Apr 17 '19 at 16:13

score 4 · Accepted Answer · edited Apr 19 '19 at 16:49

This answer gives C code because I understand C and not C#, but I think it should be pretty transferable.

By default sys.stderr writes to some console somewhere and so you can't meaningfully try to read from it. However, it's perfectly possible to replace it to redirect the output. Two sensible options include writing to a file, and writing to a StringIO object that can later be queried.

The C code to run is basically equivalent to:

import sys
from io import StringIO # Python 3
sys.stderr = StringIO()

or in C:

int setup_stderr() {
    PyObject *io = NULL, *stringio = NULL, *stringioinstance = NULL;

    int success = 0;

    io = PyImport_ImportModule("io");
    if (!io) goto done;
    stringio = PyObject_GetAttrString(io,"StringIO");
    if (!stringio) goto done;
    stringioinstance = PyObject_CallFunctionObjArgs(stringio,NULL);
    if (!stringioinstance) goto done;

    if (PySys_SetObject("stderr",stringioinstance)==-1) goto done;

    success = 1;

    done:
    Py_XDECREF(stringioinstance);
    Py_XDECREF(stringio);
    Py_XDECREF(io);
    return success;
}

You run this once at the start of your program.

To query the contents of sys.stderr you'd then do the equivalent of:

value = sys.stderr.getvalue()
encoded_value = value.encode() # or you could just handle the unicode output

In C:

char* get_stderr_text() {
    PyObject* stderr = PySys_GetObject("stderr"); // borrowed reference

    PyObject *value = NULL, *encoded = NULL;

    char* result = NULL;
    char* temp_result = NULL;
    Py_ssize_t size = 0;

    value =  PyObject_CallMethod(stderr,"getvalue",NULL);
    if (!value) goto done;
    // ideally should get the preferred encoding
    encoded = PyUnicode_AsEncodedString(value,"utf-8","strict");
    if (!encoded) goto done;
    if (PyBytes_AsStringAndSize(encoded,&temp_result,&size) == -1) goto done;
    size += 1;

    // copy so we own the memory
    result = malloc(sizeof(char)*size);
    for (int i = 0; i<size; ++i) {
        result[i] = temp_result[i];
    }

    done:
    Py_XDECREF(encoded);
    Py_XDECREF(value);

    return result;

}

There's a bit of effort spent copying the string. You might consider working directly with unicode and using PyUnicode_AsUCS4Copy.

You probably then want to look at clearing the string after you've written it, just done by replacing sys.stderr with a fresh StringIO object.

Thank you for this nice answer. I solved my problem by adopting the python script I called via C-API. However my original question was, how to solve this using the C-API. Therefore I think it's preferable to accept your answer. — TruckerCat, Apr 23 '19 at 10:42

score 0 · Answer 2 · answered Apr 18 '19 at 08:54

I could not find a way to acces sys.stderr via C-Api. But I realized that I can run python scripts via the c-api (See PyRun_String in the docs). So I am debugging now by writing sys.path to a textfile.

import sys
file = open('log.txt','w')
for path in sys.path:
    file.write(i+'\n')

How to access sys.stderr after calling PyErr_Print with python c-api

2 Answers2