Partially parse args/kwargs using the Python C Api

Question

I try to implement a Python function using the Python C API, that has a functionality similar to (this implementation is simplified and does not include things like NULL check of keywds):

def my_func(arg1, arg2 = 2, **kwargs):
  if arg2 == 1:
    # expect certain parameters to be in kwargs
  if arg2 == 2:
    # expect different parameters to be in kwargs
  if arg2 == 3:
   # call a Python function with the kwargs (whats in the kwargs is unknown)
   PythonFunc(**kwargs)

arg1 and arg2 can be both named and unnamed arguments.

The best I came up with was the following:

PyObject* my_func(PyObject* /*self*/, PyObject* args, PyObject* keywds) {
  PyObject* arg1, arg2;

  switch(PyTuple_Size(args)) {
  case 0:
    arg1 = PyDict_GetItemString(keywds, "arg1");
    if (arg2) PyDict_DelItemString(keywds, "arg1");
    else // throw TypeError

    arg2 = PyDict_GetItemString(keywds, "arg2");
    if (arg2) PyDict_DelItemString(keywds, "arg2");
    break;
  case 1:
    arg1 = PyTuple_GET_ITEM(args, 0);
    if (PyDict_GetItemString(keywds, "arg1")) // throw TypeError

    arg2 = PyDict_GetItemString(keywds, "arg2");
    if (arg2) PyDict_DelItemString(keywds, "arg2");
    break;
  case 2:
    arg1 = PyTuple_GET_ITEM(args, 0);
    if (PyDict_GetItemString(keywds, "arg1")) // throw TypeError

    arg2 = PyTuple_GET_ITEM(args, 1);
    if (PyDict_GetItemString(keywds, "arg2")) // throw TypeError
    break;
  default:
    // throw TypeError
  }

  if (...) // parse remaining keywds according to requirements
  else // call Python functions using the remaining keyword arguments
}

This can become quite extensive when the functions have more arguments. Is there a simpler way to achieve this behaviour e.g. using PyArg_ParseTupleAndKeywords?

score 1 · Accepted Answer · edited Feb 07 '21 at 12:15

You can't use PyArg_ParseTupleAndKeywords directly because it requires all the possible keyword arguments to be set. It doesn't have a "any remaining arguments are keywords" mode.

One option to use it would be to copy/move arg1 and arg2 to another dict and feed that to PyArg_ParseTupleAndKeywords:

int arg1;
int arg2=2;
char *keywords[] = {"arg1", "arg2", NULL};
PyObject* arg1arg2dict = PyDict_New(); // error check missing...
for (int i=0; keywords[i] != NULL; ++i) {
    PyObject* item = PyDict_GetItemString(kwds, keywords[i]);
    if (item) {
        PyDict_SetItemString(arg1arg2dict, keywords[i], item);
        PyDict_DelItemString(kwds, keywords[i]);
    }
}
        
if (!PyArg_ParseTupleAndKeywords(args, arg1arg2dict, "i|i:f", keywords, &arg1, &arg2)) {
    Py_DECREF(arg1arg2dict);
    return NULL;
}
Py_DECREF(arg1arg2dict);

Obviously that's a bit long but unlike your switch statement it automates well to adding more arguments.

Another option would be to write a loop instead of your switch statement. It's nothing too complicated and again would be easily expandable. In this version you'd avoid PyArg_ParseTupleAndKeywords (since you'd effectively have implemented a chunk of it)

PyObject* parsed_args[] = {NULL, NULL};
   char *keywords[] = {"arg1", "arg2", NULL};
        
   int i=0;
   for (; i < PyTuple_Size(args); ++i) {
        if (keywords[i] == NULL) {
            // run out of possible positional arguments - clean up and raise error
        }
            
        if (PyDict_GetItemString(kwds, keywords[i]) {
            // clean up and raise an error
        }
            
        parsed_args[i] = PyTuple_GET_ITEM(args, i);
        Py_INCREF(parsed_args[i]);
   }
   for (; keywords[i] != NULL; ++i) {
       PyObject* item = PyDict_GetItemString(kwds, keywords[i]);
       if (item) {
           Py_INCREF(item);
           parsed_args[i] = item;
           PyDict_DelItemString(kwds, keywords[i]);
       } else {
           // raise error, goto cleanup
       }
    }
}

The cleanup is just a simple loop of Py_XDECREF over parsed_args so can be handled by a shared goto.

The second option is what I was heading to as well. One thing that was unclear to me is, how I could use this dict with `PyObject_Call` later on. I realized, that even though the docs state, that it performs the same as `callable(*args, **kwargs)`, it simply passes through the dict. So when I modify it the original dict passed to `PyObject_Call` is modified as well. So do I need to copy the remaining keyword dict before `PyObject_Call`, or is it not allowed to modify the kwargs dict passed to my function. — maxbachmann, Dec 13 '20 at 18:02
I suspect most of the time it really doesn't matter if you modify the dict or copy it - very few programs are going to notice and it won't cause a segmentation fault or anything, just an unexpectedly modified dictionary. It'd probably be safer to make a copy though to make a copy. — DavidW, Dec 13 '20 at 18:09
I only realised this, since I call multiple functions with it and so when they modify it, the next one gets called with wrong arguments. I guess I will copy it for now to be safe. Btw in the first solution `Py_INCREF(item);` should be removed, since `PyDict_SetItemString` does not steal the reference. — maxbachmann, Dec 13 '20 at 18:15
As a follow up: `PyObject_Call` should copy the kwargs, but does not behave as documented: https://bugs.python.org/issue42629 — maxbachmann, Dec 16 '20 at 17:15

Partially parse args/kwargs using the Python C Api

1 Answers1