0

I have a list of functions that I want to execute, like in a pipeline.

pipeline = [f1, f2, f3, f4]

(Basically I compose the functions on the list see: https://mathieularose.com/function-composition-in-python/):

def run(pipeline):
     return functools.reduce(lambda f, g: lambda x: f(g(x)), pipeline, lambda x: x)

I want to observe when a function is called, when it finishes, if it fails, etc and log that to a file or a database or to a MongoDB, etc. Each of this functions is returning some python objects, so I want to use the return value and log its attributes, e.g.

If f1 returns a list I want to log f1 was completed at 23:00 on 04/22/2018. It returned a list of length 5 and so on

My problem is not about executing the functions, is about observing the beahaviour of the functions. I want that the functions are agnostic to how I code the pipeline.

I was wondering how to implement the observer pattern here. I know that the "observer pattern" sounds too much "Object Oriented", so my first idea was using decorators, but searching for a guide on this I found RxPy.

So, I am looking for a guide on how to solve this problem.

nanounanue
  • 7,942
  • 7
  • 41
  • 73
  • Show what you tried and why it does not work. Would `for f in pipeline: f()` work for example? – fferri Apr 23 '18 at 01:19
  • I don' t understand your comment. As I explain in the question I am /composing/ the functions, and that part is solved (I am using a simple modification of the link showed in the question). My problem is not about /executing/ the functions, is about /observing/ the beahaviour of the functions. I want that the functions are agnostic to how I code the pipeline – nanounanue Apr 23 '18 at 01:22
  • I just added the code. Please Read the question. I want a guidance about how to implement the observer pattern in this case – nanounanue Apr 23 '18 at 01:26

2 Answers2

1

Something like:

import time

def run(pipeline):
    def composed(x):
        for f in pipeline:
            y = f(x)
            print('{} completed on {}. it returned {}'.format(f, time.time(), y))
            x = y
        return x
    return composed

The observer pattern does not fit well functional-style.

Edit: callback based:

def run(pipeline, callback):
    def composed(x):
        for f in pipeline:
            y = f(x)
            callback(f, time.time(), y)
            x = y
        return x
    return composed
fferri
  • 18,285
  • 5
  • 46
  • 95
  • Thanks, What about using `decorators`? I really want to have the flexibility of changing to where I log the behaviour. – nanounanue Apr 23 '18 at 01:34
1

If you like decorators, you can use the following approach:

def log_finish_time(f):
    def g(*args):
        ret = f(*args)
        print('{} completed on {}'.format(f, time.time()))
        return ret
    return g

def log_return_value(f):
    def g(*args):
        ret = f(*args)
        print('{} returned {}'.format(f, ret))
        return ret
    return g

@log_finish_time
@log_return_value
def f1(x):
    return x+1

@log_finish_time
@log_return_value
def f2(x):
    return x*x

without changing your composition code.

If you want to have arguments to your decorators, you have to add another function in the middle (basically it's a function which returns a decorator):

def log_finish_time(log_prefix):
    def h(f):
        def g(*args):
            ret = f(*args)
            print('{}: {} completed on {}'.format(log_prefix, f, time.time()))
            return ret
        return g
    return h

@log_finish_time('A')
def f1(x):
    return x+1
fferri
  • 18,285
  • 5
  • 46
  • 95
  • This is great. So, If I want to have different observers, I can encapsulate those methods (`log_finish_time` and `log_return_value`) in an object? And then decorate the functions inside `run_pipeline`? Maybe getting the observer as an argument like `run_pipeline(pipeline, observer)`? – nanounanue Apr 23 '18 at 02:01
  • Either have the decorator accept the object as a parameter, or switch to the approach in my other answer, passing a callback to `run()` which gets called instead of the `print()`. – fferri Apr 23 '18 at 02:36
  • Thank you for your time and patience. Excellent answer – nanounanue Apr 23 '18 at 02:40