1

TLDR

I can't get unittest to run a test where I am trying to check that my Python C extensions calls exit(1) from stdlib.h.

The setup

I have a Python unit test and C extension which looks like the following:

The Python test

import unittest
import binding

class TestBindings(unittest.TestCase):

    def test_fail(self):
        self.assertRaises(SystemExit, binding.fail)

if __name__ == '__main__':
    unittest.main()

The C extension

core.h

// The main core C library declarations. 
void fail(void);

core.c

// The main core C library implementations.
void fail(void)
{
    fprintf(stderr, "We are failing now.\n");
    exit(1);
}

bindings.c

// The main core C library bindings.
#include <Python.h>
#include <stdio.h>
#include <stdlib.h>

#include "core.h"

PyObject *_fail(PyObject *self, PyObject *args, PyObject *kwargs)
{
    fail();
    Py_RETURN_NONE;
}

static PyMethodDef binding_methods[] = {
        {"fail", PyFunc(_fail), METH_VARARGS | METH_KEYWORDS,
         "Fails and calls exit()."},
        {NULL, NULL, 0, NULL} /* Sentinel */
};

static struct PyModuleDef binding_module = {
        PyModuleDef_HEAD_INIT,
        "binding",                                    
        "A simple module to demonstrate C bindings.", 
        -1,                                           
        binding_methods};

PyMODINIT_FUNC
PyInit_binding(void)
{
    return PyModule_Create(&binding_module);
}

The tests don't run

The Python test always exits, and does not run the remaining tests nor report the results. I have tried to find ways to capture this exiting behaviour, but can't find anything that works.

My thoughts are to either:

  • Add a more robust capture in the unit test.
  • Add/declare some form of error handler in the module containing all the python bindings.
  • Change the way the core library exits when something goes wrong and replace all my calls to exit(1). I want to avoid any changes to the core* files.

EDIT

After some more digging around, there were two routes that looked promising:

  1. Trying to add functionality when exit is called (without trying to mock/stub the C standard library exit). This led me to functions such as atexit (and the even nicer GNU extension on_exit). However, I couldn't figure out a way to add my custom exit without calling another exit from within exit, and calling exit twice is undefined behaviour.

  2. Launching the test in its own process which is allowed to die, and then checking the return code for this other process. This is the closest to the requirements I had, and seems to solve my problem with minimal code changes in the core C library, and has a solution outlined here.

oliversm
  • 1,771
  • 4
  • 22
  • 44
  • I'm sorry but why do you want to unit test such a simple function? It literally only prints something to stderr, then exits. – Marco Jul 27 '23 at 18:00
  • This is just a simple case, but represents a more general pattern. My core C library calls `exit()` when it wants to exit, and in my Python unit tests I want to check that certain conditions cause the library to abort. At the moment I am struggling to catch this. In my more extensive functions to can replace `fail()` with `some_more_complicated_function_which_might_fail()`. – oliversm Jul 27 '23 at 18:07
  • 3
    The `exit()` function does what it says - causes the entire process to exit. Python interpreter, test suite, everything. If you want to "test" some code with an `exit()` in any meaningful way you'll need some kind of mocking system (possibly one that links a testing library with a replacement `exit()` at test time?) Python C extensions run in the same process as the interpreter, this isn't IPC / calling a separate program. That said, I can't think of ANY valid use case for a Python extension to call `exit()`. You may wish to meditate on your requirements and design. – BadZen Jul 27 '23 at 18:08
  • 2
    Then you have to stub out `exit()` and replace it with something else during testing. As @BadZen said _"The exit() function does what it says - causes the entire process to exit."_ – Marco Jul 27 '23 at 18:10
  • @BadZen - I can see that `exit()` was designed with consideration of other processes in mind. However the remark "I can't think of ANY valid use case for a Python extension to call `exit()`" I can't agree with. Many functions in C libraries call `exit` (or `abort`) or similar. If these are being backfitted and wrapped up with C extensions, then it is quite natural that a Python extension would call `exit()`. – oliversm Jul 27 '23 at 18:24
  • 1
    This isn't really a place to have this discussion, but I strongly disagree. Part of the job of "wrapping" this functionality and adapting it to Python is to make it work smoothly in that context. It's frankly bad design to have a python extension ever call `exit()`, and I anticipate that would not be an even slightly controversial opinion among skilled developers. Happy to explain in more detail in PM / side chat if you like. – BadZen Jul 27 '23 at 18:28
  • Just in addition to this, in the C library tests I am able to test for a call to `exit`. E.g. in the Criterion testing framework, I can write `Test(logging, test_exit, .exit_code = 1 ){ exit(1); }` and the test runs fine. I was expecting to be able to "catch" a call to exit almost equally well from within Python. – oliversm Jul 27 '23 at 18:30
  • 3
    Not a python developer but I strongly agree with _"It's frankly bad design to have a python extension ever call exit()"_. IMO it's only warranted to call `exit()` (or alike) in _extreme_ cases where program behaviour cannot be guaranteed from that point onwards (an UNEXPECTED buffer overflow may be a good example for an extreme case). This is an assumption made by me but you "unit testing" this kind of behaviour implies that it is expected and part of the API - which IMO _is_ bad API design. I agree with @BadZen and suggest you may want to re-think your design. – Marco Jul 27 '23 at 18:57
  • 2
    Your expectation is wrong. And I have no idea what the Criterion testing framework is but that doesn't look like valid C and I suspect the `exit(1)` in that example is not calling the libc exit function. Code meant to be used as a library should **never** call `exit()`. – Kurtis Rader Jul 27 '23 at 18:58
  • 1
    @oliversm *Many functions in C libraries call exit (or abort) or similar.* Not any that someone concerned with reliability and availability will **EVER** use. **Library functions NEVER get to decide when to obliterate the entire process and potentially cause data loss**. Tell your customers, "Yeah, whenever **I** think it's appropriate, **I** will decide to kill **your** entire process." Tell us how they respond. – Andrew Henle Jul 27 '23 at 19:07
  • 1
    @AndrewHenle - Very nice point about who is and isn't allowed to call `exit` and the boundary between the library code and the application code. – oliversm Jul 27 '23 at 19:21
  • Some library functions *do* call exit (or otherwise terminate the process). These are used exclusively for process control (most often in the context of a a command-line tool top-level) and it is nuclear-bad to call this from an *API*. In practice, this means the C library, or the libraries that *implement your runtime* are the only places such calls should be made. – BadZen Jul 29 '23 at 03:21
  • The correct approach, in this case, is to signal an exception (or use some other error mechanism) from the C code and exit **cleanly**, if appropriately, via the runtime's mechanism for doing so. This is how data gets lost from files or databases. – BadZen Jul 29 '23 at 03:21

1 Answers1

0

If you have any control whatsoever over the c code, you could change the exit() into a callback into python, and the callback does say.exit(). You can then mock this python callback to verify that it’s called.

Otherwise, I think you’re out of luck. Python can’t stub out C code.

Frank Yellin
  • 9,127
  • 1
  • 12
  • 22