Autodiscover python decorators

Question

I was wondering if there is a standardized approach or best practice to scan/ autodiscover decorators like it is done here but also in several other libs like Django, Flask. Usually a decorator provides extra/ wrapped functionality right at the time the inner func is called.

In the example shown below but also in Flask/ Django (route decorators) the decorator is rather used to add overarching functionalities, e.g. spawning of a tcp client initially within the decorator logic and then call the inner func when there is a message received to process it.

Flask/ Django register for an url route where the inner func is only called later when the url is requested. All examples require an initial registration (scan/ discover) of decorator logic to also initially start the overarching functionality. To me this seems to be an alternative use of decorators and I would like to understand the best practice approach if there is any.

See Faust example below where decorator app.agent() automatically triggers a listening (kafka stream) client within asyncio event loop and incoming message is then processed by the inner function hello() later, only when there is a message received, requiring an initial check/ scan/ discovery of related decorator logic first at the start of the script.

import faust

class Greeting(faust.Record):
    from_name: str
    to_name: str

app = faust.App('hello-app', broker='kafka://localhost')
topic = app.topic('hello-topic', value_type=Greeting)

@app.agent(topic)
async def hello(greetings):
    async for greeting in greetings:
        print(f'Hello from {greeting.from_name} to {greeting.to_name}')

@app.timer(interval=1.0)
async def example_sender(app):
    await hello.send(
        value=Greeting(from_name='Faust', to_name='you'),
    )

if __name__ == '__main__':
    app.main()

Any hints to update this question are welcome as I assume I am just searching for the wrong keywords here.. — trbck, Nov 22 '18 at 20:40
I'm not too hot on decorator knowledge, but I don't understand what you mean by "discover". — roganjosh, Nov 22 '18 at 20:45
app.main() is collecting/ discovering/ searching for all decorators to provide additional functionalities. Flask is doing something very similar, see first code example here: http://flask.pocoo.org/ — trbck, Nov 22 '18 at 20:56
Yeah! Faust uses "venusian" to search for decorators. This will actually recursively import all the modules in your package directory. I'm not a huge fan of this approach, but it is convenient. The caveat is it can easily import something that it shouldn't, such as a test module or a script that is not meant for being imported. Django uses a different approach, there you explicitly list INSTALLED_APPS and it searches for a "admin.py" module in those installed apps. — asksol, Dec 10 '18 at 01:50

roganjosh · Answer 1 · 2018-11-22T21:23:34.997

2

Nothing is "discovered". When you import a module from a package, all of that code is executed. This is why we have if __name__ == '__main__' to stop certain code being executed on import. The decorators will be "discovered" when you run your code.

I think the Flask blueprint is a nice example. Here you can see how it registers the url endpoints when you import modules. All it's doing is appending to a list:

    def route(self, rule, **options):
        """Like :meth:`Flask.route` but for a blueprint.  The endpoint for the
        :func:`url_for` function is prefixed with the name of the blueprint.
        """
        def decorator(f):
            endpoint = options.pop("endpoint", f.__name__)
            self.add_url_rule(rule, endpoint, f, **options)
            return f
        return decorator

The code runs, the decorators are evaluated and they need only keep some internal list of all the functions they decorate. These are stored in the Blueprint object.

edited Nov 22 '18 at 21:23

answered Nov 22 '18 at 21:14

roganjosh

12,594
4
29
46

Thanks. According to your example the code would run on import to add the endpoint rule but also start the inner function as well which would not be the expected outcome I guess. Please see https://github.com/pallets/flask/blob/dcc02d6e7d4bf486e64aa5b6e55a75501c2ba2e5/flask/blueprints.py#L134 as well as https://github.com/robinhood/faust/blob/master/faust/app/base.py#L531 - after further searching I came across https://docs.pylonsproject.org/projects/venusian/en/latest/ (noted in the faust repo). I think this is what I was looking for. However, this seems complicated.. – trbck Nov 23 '18 at 01:40
@trbck I think you need to be more clear here on your expected behaviour. The first of your links is no different than what I have dug out. The others seems meaningless to me without more context from your side. Are you _sure_ you need something more complex or have you settled on some idea you might not even need? – roganjosh Nov 23 '18 at 09:12
Agree, thanks. Usually a decorator provides extra/ wrapped functionality when the inner func is called. In all mentioned examples the decorator is rather used to add overarching functionalities, e.g. faust spawns a kafka streams listener initially and then calls inner func when there is a message received. Flask/ Django register for an url route and then the inner func is only called later when the url is requested. All examples require an initial registration (discover) to also initially start the overarching functionality. To me this seems to be an alternative use of decorators... – trbck Nov 23 '18 at 22:08

Autodiscover python decorators

1 Answers1