Allow help() to work on partial function object

Question

I'm trying to make sure running help() at the Python 2.7 REPL displays the __doc__ for a function that was wrapped with functools.partial. Currently running help() on a functools.partial 'function' displays the __doc__ of the functools.partial class, not my wrapped function's __doc__. Is there a way to achieve this?

Consider the following callables:

def foo(a):
    """My function"""
    pass

partial_foo = functools.partial(foo, 2)

Running help(foo) will result in showing foo.__doc__. However, running help(partial_foo) results in the __doc__ of a Partial object.

My first approach was to use functools.update_wrapper which correctly replaces the partial object's __doc__ with foo.__doc__. However, this doesn't fix the 'problem' because of how pydoc.

I've investigated the pydoc code, and the issue seems to be that partial_foo is actually a Partial object not a typical function/callable, see this question for more information on that detail.

By default, pydoc will display the __doc__ of the object type, not instance if the object it was passed is determined to be a class by inspect.isclass. See the render_doc function for more information about the code itself.

So, in my scenario above pydoc is displaying the help of the type, functools.partial NOT the __doc__ of my functools.partial instance.

Is there anyway to make alter my call to help() or functools.partial instance that's passed to help() so that it will display the __doc__ of the instance, not type?

I don't think there is any way around this; there are no hooks we can grab here. — Martijn Pieters, May 21 '13 at 14:51

durden2.0 · Accepted Answer · 2013-05-21T18:37:59.113

I found a pretty hacky way to do this. I wrote the following function to override the __builtins__.help function:

def partialhelper(object=None):
    if isinstance(object, functools.partial):
        return pydoc.help(object.func)
    else:
        # Preserve the ability to go into interactive help if user calls
        # help() with no arguments.
        if object is None:
            return pydoc.help()
        else:
            return pydoc.help(object)

Then just replace it in the REPL with:

__builtins__.help = partialhelper

This works and doesn't seem to have any major downsides, yet. However, there isn't a way with the above naive implementation to support still showing the __doc__ of some functools.partial objects. It's all or nothing, but could probably attach an attribute to the wrapped (original) function to indicate whether or not the original __doc__ should be shown. However, in my scenario I never want to do this.

Note the above does NOT work when using IPython and the embed functionality. This is because IPython directly sets the shell's namespace with references to the 'real' __builtin__, see the code and old mailing list for information on why this is.

So, after some investigation there's another way to hack this into IPython. We must override the site._Helper class, which is used by IPython to explicitly setup the help system. The following code will do just that when called BEFORE IPython.embed:

import site
site._Helper.__call__ = lambda self, *args, **kwargs: partialhelper(*args, **kwargs)

Are there any other downsides I'm missing here?

It's also worth noting this will NOT work in IPython. I'm sure there's a way around it, but IPython doesn't have `__builtins__.help` even though the `help()` function is available in the global namespace. Suggestions? — durden2.0, May 21 '13 at 15:44

score 0 · Answer 2 · answered May 21 '13 at 15:01

0

how bout implementing your own?

def partial_foo(*args):
    """ some doc string """
    return foo(*((2)+args))

not a perfect answer but if you really want this i suspect this is the only way to do it

answered May 21 '13 at 15:01

Joran Beasley

110,522
12
160
179

This will work, but not very well for my scenario. I have a big list of functions that all need to have the same argument defaulted to something at runtime. So, this would require a bunch of these wrapper functions. – durden2.0 May 21 '13 at 15:04

Jake Stevens-Haas · Answer 3 · 2020-10-01T19:33:50.410

You identified the issue - partial functions aren't typical functions, and the dunder variables don't carry over. This applies not just to __doc__, but also __name__, __module__, and more. Not sure if this solution existed when the question was asked, but you can achieve this more elegantly ("elegantly" up to interpretation) by re-writing partial() as a decorator factory. Since decorators (& factories) do not automatically copy over dunder variables, you need to also use @wraps(func):

def wrapped_partial(*args, **kwargs):
    def foo(func):
        @wraps(func)
        def bar(*fargs,**fkwargs):
            return func(*args, *fargs, **kwargs, **fkwargs)
        return bar
    return foo

Usage example:

@wrapped_partial(3)
def multiply_triple(x, y=1, z=0):
    """Multiplies three numbers"""
    return x * y * z

# Without decorator syntax: multiply_triple = wrapped_partial(3)(multiply_triple)

With output:

>>>print(multiply_triple())
0
>>>print(multiply_triple(3,z=3))
9
>>>help(multiply_triple)

help(multiply_triple)
Help on function multiply_triple in module __main__:

multiply_triple(x: int, y: int = 1, z: int = 0)
    Multiplies three numbers

Thing that didn't work, but informative when using multiple decorators

You might think, as I first did, that based upon the stacking syntax of decorators in PEP-318, you could put the wrapping and the partial function definition in separate decorators, e.g.

def partial_func(*args, **kwargs):
    def foo(func):
        def bar(*fargs,**fkwargs):
            return func(*args, *fargs, **kwargs, **fkwargs)
        return bar
    return foo

def wrapped(f):
     @wraps(f)
     def wrapper(*args, **kwargs):
         return f(*args, **kwargs)
     return wrapper

@wrapped
@partial_func(z=3)
def multiply_triple(x, y=1, z=0):
    """Multiplies three numbers"""
    return x * y * z

In these cases (and in reverse order), the decorators are applied one at a time, and the @partial_func interrupts wrapping. This means that if you are trying to use any decorator that you want to wrap, you need to rewrite the decorator in a factory where the decorator's return function is itself decorated by @wraps(func). If you are using multiple decorators, they all have to be turned into wrapped factories.

Alternate method to have decorators "wrap"
Since decorators are just functions, you can write a copy_dunder_vars(obj1, obj2) function that retruns obj2 but with all the dunder variables from obj1. Call as:

def foo()
    pass

foo = copy_dunder_vars(decorator(foo), foo)

This goes against the preferred syntax, but practicality beats purity. I think "not forcing you to rewrite decorators that you're borrowing from elsewhere and leaving largely unchanged" fits into that category. After all that wrapping, don't forget ribbon and a bow ;)

Why not just use the "roughly equivalent" implementation mentioned in the [docs](https://docs.python.org/3/library/functools.html?highlight=functools%20wraps#functools.partial)? Decorating the function you want to partially apply at definition time defeats the purpose; you should just use a constant inside the function. — Chris Wesseling, Sep 30 '20 at 08:25
Good point. I guess OP's use case wouldn't use decorator syntax, and would more likely use it as `foo=wrapped_partial(args)(bar)`. Also, that example doesn't include copying over `__docs__`, `__name__`, &c. You can add those in of course, but to make sure you get all the dunders (including those that may be added in future pythons), you may as well use `@wraps` — Jake Stevens-Haas, Oct 01 '20 at 19:31
Not really. `wraps` copies the `__module__` and `__name__` from the function you pass in, to prevent all decorated functions to have the name of the decorator. In other words: what comes out of a decorator should "be" what goes in, but with changed behaviour. `partial` creates a new function, with a different number of arguments and a different name. So the `__name__` and `__module__` should not be of what goes in, but those of the variable it gets assigned to (the caller). If you really want to do that in your implementation of `partial` you could hack with `inspect` to get the calling frame. — Chris Wesseling, Oct 02 '20 at 16:27
There are probably use cases for both patterns, and OP's stance on `__name__` and `__module__` will remain a mystery. I ended up on this question because my use cases called for wrapping all the dunders. Though I would be grateful if you can point out cases when wrapping `__module__` and `__name__` can cause problems (`%autoreload?`). — Jake Stevens-Haas, Oct 04 '20 at 19:15
To wit: Custom loss functions in Keras need to accept two arguments, `y_pred` and `y_true`. However, a Generator's loss in a GAN depends on the Discriminator. I wrote the functions `early_gen_loss()` and `late_gen_loss()` in `losses.py` to accept the two arguments Keras needs plus the Discriminator. When I build the GAN, I use `wrapped_partial()` to pass the Discriminator. I still want the function to be called `early_gan_loss()` and it should know it's from the `losses` module. I suppose a factory function might also work, but I sometimes want to call the loss with all three arguments. — Jake Stevens-Haas, Oct 04 '20 at 19:19

Allow help() to work on partial function object

3 Answers3

Linked