Add a signature, with annotations, to extension methods

Question

When embedding Python in my application, and writing an extension type, I can add a signature to the method by using a properly crafted .tp_doc string.

static PyMethodDef Answer_methods[] = {
  { "ultimate", (PyCFunction)Answer_ultimate, METH_VARARGS, 
    "ultimate(self, question='Life, the universe, everything!')\n"
    "--\n"
    "\n"
    "Return the ultimate answer to the given question." },
  { NULL }
};

When help(Answer) is executed, the following is returned (abbreviated):

class Answer(builtins.object)
 |
 |  ultimate(self, question='Life, the universe, everything!')
 |      Return the ultimate answer to the given question.

This is good, but I'm using Python3.6, which has support for annotations. I'd like to annotate question to be a string, and the function to return an int. I've tried:

static PyMethodDef Answer_methods[] = {
  { "ultimate", (PyCFunction)Answer_is_ultimate, METH_VARARGS, 
    "ultimate(self, question:str='Life, the universe, everything!') -> int\n"
    "--\n"
    "\n"
    "Return the ultimate answer to the given question." },
  { NULL }
};

but this reverts to the (...) notation, and the documentation becomes:

 |  ultimate(...)
 |      ultimate(self, question:str='Life, the universe, everything!') -> int
 |      --
 |
 |      Return the ultimate answer to the given question.

and asking for inspect.signature(Answer.ultimate) results in an exception.

Traceback (most recent call last):
  File "<string>", line 11, in <module>
  File "inspect.py", line 3037, in signature
  File "inspect.py", line 2787, in from_callable
  File "inspect.py", line 2266, in _signature_from_callable
  File "inspect.py", line 2090, in _signature_from_builtin
ValueError: no signature found for builtin <built-in method ultimate of example.Answer object at 0x000002179F3A11B0>

I've tried to add the annotations after the fact with Python code:

example.Answer.ultimate.__annotations__ = {'return': bool}

But the builtin method descriptors can't have annotations added this way.

Traceback (most recent call last):
  File "<string>", line 2, in <module>
AttributeError: 'method_descriptor' object has no attribute '__annotations__'

Is there a way to add annotations to extension methods, using the C-API?

---

Argument Clinic looked promising and may still be very useful, but as of 3.6.5, it doesn't support annotations.

annotation
The annotation value for this parameter. Not currently supported, because PEP 8 mandates that the Python library may not use annotations.

Answer 1

TL;DR There is currently no way to do this.

How do signatures and C extensions work together?

In theory it works like this (for Python C extension objects):

• If the C function has the "correct docstring" the signature is stored in the __text_signature__ attribute.
• If you call help or inspect.signature on such an object it parses the __text_signature__ and tries to construct a signature from that.

If you use the argument clinic you don't need to write the "correct docstring" yourself. The signature line is generated based on comments in the code. However the 2 steps mentioned before still happen. They just happen to the automatically generated signature line.

That's why built-in Python functions like sum have a __text-signature__s:

>>> sum.__text_signature__
'($module, iterable, start=0, /)'

The signature in this case is generated through the argument clinic based on the comments around the sum implementation.

What are the problems with annotations?

There are several problems with annotations:

• Return annotations break the contract of a "correct docstring". So the __text_signature__ will be empty when you add a return annotation. That's a major problem because a workaround would necessarily involve re-writing the part of the CPython C code that is responsible for the docstring -> __text_signature__ translation! That's not only complicated but you would also have to provide the changed CPython version so that it works for the people using your functions.

  Just as example, if you use this "signature":

  ultimate(self, question:str='Life, the universe, everything!') -> int
  

  You get:

  >>> ultimate.__text_signature__ is None
  True
  

  But if you remove the return annotation:

  ultimate(self, question:str='Life, the universe, everything!')
  

  It gives you a __text_signature__:

  >>> ultimate.__text_signature__
  "(self, question:str='Life, the universe, everything!')"
  

• If you don't have the return annotation it still won't work because annotations are explicitly not supported (currently).

  Assuming you have this signature:

  ultimate(self, question:str='Life, the universe, everything!')
  

  It doesn't work with inspect.signature (the exception message actually says it all):

  >>> import inspect
  >>> inspect.signature(ultimate)
  Traceback (most recent call last):
  ...
      raise ValueError("Annotations are not currently supported")
  ValueError: Annotations are not currently supported
  

  The function that is responsible for the parsing of __text_signature__ is inspect._signature_fromstr. In theory it could be possible that you maybe could make it work by monkey-patching it (return annotations still wouldn't work!). But maybe not, there are several places that make assumptions about the __text_signature__ that may not work with annotations.

Would PyFunction_SetAnnotations work?

In the comments this C API function was mentioned. However that deliberately doesn't work with C extension functions. If you try to call it on a C extension function it will raise a SystemError: bad argument to internal function call. I tested this with a small Cython Jupyter "script":

%load_ext cython

%%cython

cdef extern from "Python.h":
    bint PyFunction_SetAnnotations(object func, dict annotations) except -1

cpdef call_PyFunction_SetAnnotations(object func, dict annotations):
    PyFunction_SetAnnotations(func, annotations)

>>> call_PyFunction_SetAnnotations(sum, {})

---------------------------------------------------------------------------
SystemError                               Traceback (most recent call last)
<ipython-input-4-120260516322> in <module>()
----> 1 call_PyFunction_SetAnnotations(sum, {})

SystemError: ..\Objects\funcobject.c:211: bad argument to internal function

So that also doesn't work with C extension functions.

Summary

So return annotations are completely out of the question currently (at least without distributing your own CPython with the program). Parameter annotations could work if you monkey-patch a private function in the inspect module. It's a Python module so it could be feasible, but I haven't made a proof-of-concept so treat this as a maybe possible, but probably very complicated and almost certainly not worth the trouble.

However you can always just wrap the C extension function with a Python function (just a very thing wrapper). This Python wrapper can have function annotations. It's more maintenance and a tiny bit slower but saves you all the hassle with signatures and C extensions. I'm not exactly sure but if you use Cython to wrap your C or C++ code it might even have some automated tooling (writing the Python wrappers automatically).