Python C extension decorator

Question

I apologize in advance for the long-winded background info.

I've been playing around with the Python/C-API recently (Python 3.4) and have gotten stumped. My goal is to have a C extension which I can use as a function/method decorator. I have a reasonably well working prototype for an LRU cache in the gist, https://gist.github.com/pbrady/916495198910e7d7c713. Although the cache works and is very fast, there are two issues problems :

1. The type returned by the my decorater is not function: i.e.

   >>> from lrucache import lrucache
   >>> @lrucache()
   ... def f(a, b):
   ...    return a+b
   ...
   >>> type(f)
   >>> <class 'lrucache.cache'>
   

   Because of this, the decorator does not work properly on methods - it seems that self gets lost because Python doesn't make an instance method when it sees my class (which makes sense).

2. Copying __doc__ from the decorated function to my cache class doesn't impact the message displayed by help.

My idea to get around these issues was to simply return the user function with a few modifications rather than a new custom class object. These modifications are

1. Add a __wrapped__ attribute to the function and point it to the function.

2. Overwrite the __call__ attribute so that function calls are directed to a custom C routine.

I was able to accomplish (1) but overriding a functions __call__ method doesn't do anything since it's not generally used by the interpreter.

My Question (finally): How do I create a Python function (i.e., a PyFunctionObject) which will call a C function?

Or, if there is a better way to do this, I would be interested in that as well. This is largely a learning/fun exercise so I'm not too interested in Cython based solutions.

Answer 1

Well your (2) step obviously fails because special attributes are looked up on the class, not on the instance:

In [1]: class Test:
   ...:     def __call__(self):
   ...:         print('Called class attribute!')
   ...:         

In [2]: t = Test()

In [3]: def new_call():
   ...:     print('Called instance attribute!')
   ...:     

In [4]: t.__call__ = new_call

In [5]: t()
Called class attribute!

And you don't want to modify the function class __call__ method, since this would modify the semantics of the python code in a substantial way.

As far as I know there is one way to instantiate a PyFunctionObject, which is calling PyFunction_New. The problem with it is that it requires a PyCodeObject as argument and to create such an object you can call:

• PyCode_NewEmpty: creates an invalid code object. It is used only to create frame objects where you don't care of the code to put there.
• PyCode_New: creates a bytecode object. This is a beast with 14 arguments, one of which is a PyObject *code which should be a readable buffer that contains the binary representation of the bytecode.
• Py_CompileStringObject: this seems the only reasonable solution. Basically this is the compile built-in.

Once you have created the code object the creation of the function is trivial. Note however that you wont gain any performance benefit doing this since the body of the function object is interpreted.

However there are other solutions to your problem:

• You could make your lrucache a descriptor. This is what python does for methods to automatically pass self as first parameter, so you can emulate that behaviour.

• You could write a very very simple python module that imports your C extension and provides a function such as:

  from functools import wraps
  
  from _lrucache import cache
  
  def lrucache(func):
      cached_func = cache(func)
  
      @wraps(func)
      def wrapper(*args, **kwargs):
          return cached_func(*args, **kwargs)
  
      return wrapper
  

  In this way you side-step the whole problem via python code.

• You could create a subclass of the function type and return that object instead. However this might not work if python is checking for the exact type when building a class, and the implementation might not be trivial since the interpreter might assume some properties about function objects that you'd have to provide. No, you can't subclass a function...