invoking yield for a generator in another function

Question

suppose I have some manager object. This object's API has a main_hook function, that gets another function f as it's argument, and runs the given f in a loop, doing some stuff in between each iteration:

def main_hook(self,f):
    while (self.shouldContinue()):
        #do some preparations
        f(self)
        #do some tear down

Now, I also have (more accurately, would like to have) a function stop_and_do_stuff, that once called, stops main_hook dead in it's tracks, returns the control to whichever func called main_hook, and after that func finished what's it doing, get control back to main_hook and continue. Basically the result will be the same as doing

def main_hook(self,f):
    while (self.shouldContinue()):
        #do some preparations
        yield
        #do some tear down

Except that instead yield I want to have a call to f(), while giving f the option to call self.stop_and_do_stuff()

I can't work around this by making f also a generator for 2 reasons:

1.f isn't part of my API - it's given to me by a user who uses my lib

2.Even if could ask him to use yield, the place in the code in which he will need to call stop_and_do_stuff won't be directly inside f, rather in some place in the function stack which will be inside f(), but not directly in it, e.g

def h(manager):
    #do stuff
    if should stop:
        manager.stop_and_do_stuff()
    #do more stuff
def g(manager):
    #some stuff
    if should stop:
        manager.stop_and_do_stuff()
    #more stuff
    if should stop again:
        manager.stop_and_do_stuff()  
    if should call h:
        h()
def f(manager):
    g(manager)

so if I choose to make f a generator, I also need to make g a generator and also h, otherwise this trick won't work.

Is there any solution to all of this? maybe I'm trying to solve it the wrong way?

(I know this question is long and ugly - it's the best I could do. If something isn't clear please tell me and I'll clarify it)

EDIT

Maybe pep 342 is the solution?

Answer 1

My previous answer describes how to do this in Python2, which is very ugly. But now I ran across PEP 380: Syntax for Delegating to a Subgenerator. That does exactly what you ask. The only problem is that it requires Python3. But that shouldn't really be a problem.

Here's how it works:

def worker():
    yield 1
    yield 2
    return 3

def main():
    yield 0
    value = yield from worker()
    print('returned %d' % value)
    yield 4

for m in main():
    print('generator yields %d' % m)

The result of this is:

generator yields 0
generator yields 1
generator yields 2
returned 3
generator yields 4

Exceptions are passed through the way you would expect.

Answer 2

I believe I should also add an answer from the other point of view, ie not trying to explain how you could achieve what we can understand of what you are trying to do, but why yield definitely couldn't possibly work.

When a function contains yield keyword it is deeply modified. It is still a callable but not a normal function any more : it becomes a factory that return an iterator.

From the caller's point of view there is no difference between the three implementations below (except that the yield one is so much simpler).

##########################################
print "Function iterator using yield",

def gen():
    for x in range(0, 10):
        yield x

f = gen()
try:
    while True:
        print f.next(),
except StopIteration:
    pass

for x in gen():
    print x,

print

#########################################
print "Class iterator defining iter and next",

class gen2(object):

    def __init__(self):
        self.index = 0;
        self.limit = 10;

    def __iter__(self):
        return self

    def next(self):
        if self.index >= self.limit:
            raise StopIteration
        self.index += 1;
        return self.index - 1;


f = gen2()
try:
    while True:
        print f.next(),
except StopIteration:
    pass

for x in gen2():
    print x,
print

#########################################
print "Function iterator using iter() and sentinel",
def gen3():
    def g3():
        if g3.index is None:
            g3.index = 0
        g3.index += 1;
        return g3.index - 1

    g3.index = None
    return iter(g3, 10)

f = gen3()
try:
    while True:
        print f.next(),
except StopIteration:
    pass

for x in gen3():
    print x,
print

Then you should understand that yield is not much about control flow, but about keeping call context inside variables. Once it is understood you have to decide if the API of main_loop really want to provide an iterator to it's caller. Then if so, if f may loop it must should also be an iterator (and there should be a loop around calls to f() like below).

def main_hook(self,f):
    while (self.shouldContinue()):
        #do some preparations
        for v in f(self):
            yield v
        #do some tear down

But you should not care if f() has to call inner functions g(), etc. That is completely irrelevant. You provide a lib and it is your user problem to call with an appropriate iterable. If you believe your lib user won't be able to, you will have to change the overall design.

Hope it helps.

Answer 3

I don't understand the whole either (what does the main_hook caller look like ?), but i would say, Throw a StopNow exception, when you should stop, just like you should throw StopIteration when your generator is finished.

here is how i understood the thing as well as what i would do.

class StopNow(Exception):
    pass

def main_hook(self,f):
    got_stop_now_exc = False
    while (!got_stop_now_exc and self.shouldContinue()):
        #do some preparations
        try:
             f(self)
        except StopNow:
             got_stop_now_exc = True

        #do some compulsary tear down, exception or not

def stop_and_do_stuff()
    raise StopNow()
def my_f():
    if needed:
        stop_and_do_stuff()

def the_main_hook_caller():
    while i_should:
        managerthingie.main_hook(my_f)
        do_stuff()

Answer 4

I am not quite sure what exactly you are trying to achieve, so maybe if you can explain the problem more instead of giving solution that would be better.

From my partial understanding why don't you do something like this

def main_hook(self,f):
    while (self.shouldContinue()):
        #do some preparations
        stop_and_do_stuff = f(self)
        if stop_and_do_stuff :
            yield
            
        #do some tear down
    

So basically f returns a flag to stop or not, and if it says stop we yield to function which called main_hook and that function can continue after doing some stuff

e.g.

class A(object):
    def main_hook(self,f):
        while (self.shouldContinue()):
            #do some preparations
            stop = f(self)
            if stop:
                yield
                
            #do some tear down
        
    def shouldContinue(self):
        return True
    
def f(a):
    return True

a = A()
for x in a.main_hook(f):
    print x

Answer 5

The behavior you describe looks exactly like a simple function call. Like below.

def f(manager):
    print("Entering f")
    manager.stop_and_do_stuff()
    print("Exiting f")

class Manager(Object):
    def shouldContinue(self):
        return True

    def stop_and_do_stuff(self):
        print("Manager stop and do stuff")

    def main_hook(self,f):
        while self.shouldContinue()
            print("Manager Setup")
            f(self)
            print("Manager Tear Down")

No problem if f() is provided by another user of if stop_and_do_stuff is called from some inner function. If you also want the manager to be able to unwind stack from stop_and_do_stuff and really exit in some cases, no problem. Just raise some exception from it and you would catch it from main_hook or upper code.

You should be able to do from inside stop_and_and_do_stuff() whatever you want to do from the caller of main hook. If not you should explain why.

What is unclear in the question is what's happening on the caller side of main_hook() and why you would want to be able to exit the main_hook loop, but not really. Either the main_loop caller expect a generator either it does not. You need to explain that part if you want to get a sensible answer (some context informations would also be nice, if you really explain WTF you are trying to do, and your real restrictions - you said f is provided by some other user and main_hook is in a lib, what of main_hook caller ? - there is probably well known usual solutions).