Abstracting away boiler plate sqlalchemy call to allow different options in a try/except block

Question

I'm trying to abstract my sqlalchemy query call, the complete unmodified call looks like this:

Session.query(User).options(FromCache('redis1')).filter(User.id == user_id).all()

The dynamic parts I need control over is User, redis1 and the last call all() I want to call something like above line with the possibility to change the things in bold above in the arguments.

My first try turned out like this:

# Abstracts the final chained method
def q(obj, action):
    return getattr(obj, action)()

# Removes a lot of boiler plate code
# but doesn't give me control over the cache argument
# which I need in the same method as the action above
def query(model):
    return Session.query(model).options(FromCache('redis1'))

# Called like this
result = q(query(User).filter(User.id == user_id), 'all')

But I'd obviously only want one function instead of two so I can control both the action and the model. The reason for this is that I'd like to wrap the .one() or .all() call in a try-except block that will go through different cache servers in the FromCache option in case the query fails. But doing this every time I write a query would be annoying.

I'm looking for something like this (code doesn't work, obviously):

def q(model, filters, action):
    data = None
    servers = ['redis1', 'redis2']
    for server in servers:
        try:
            call = Session.query(model).options(FromCache(redis)).filters_here_somehow()
            data = getattr(call, obj)() # Final call from action added
        except:
            continue
        break;
    return data

# and called something like this.. (or some other way)
result = q(User, the filter/between/limit/whatnot here, 'all')

Any ideas or am I totally way off base? Or is there a smarter way to do this?

zzzeek · Accepted Answer · 2013-01-31T17:37:07.690

the only part of this where something interesting has to happen is the "all()". The rest of it seems like you'd stick with a straight Query.

So here's one way pattern:

q = query(User).filter(User.id = user_id).limit(whatever)
results = from_redis(q, Query.all)

and here's a way we can do that:

def from_redis(query, meth):
    for server in ['r1', 'r2']:
        try:
            return meth(q.options(FromCache(server)))
        except Exception, e:
            continue
    else:
        raise e

but that might seem a little lame. Since you're using the caching recipe and subclassing Query, you could add your redis feature to it as well, sort of like:

class RedisQuery(CachingQuery):
    redis_servers = False

    @_generative(self):
    def using_redis(self):
        self.redis_servers = True

    def __iter__(self):
        if self.redis_servers:
            for server in ['r1', 'r2']:
                try:
                    return super(RedisQuery, self).options(FromCache(server)).__iter__()
                except Exception, e:
                    continue
            else:
                raise e
        else:
            return super(RedisQuery, self).__iter__()

that one you call like this:

q = query(User).filter(User.id = user_id).limit(whatever)
results = q.using_redis().all()

just some ideas.

Thanks for your reply. I'm interested in your second example but as per your dogpile example there's already a subclass of query containing an __ iter __ method. Here's the query-class from the dogpile example: http://pastebin.com/6RV7Mi6y. Could you please show me how to integrate your answer in that code? :) — moodh, Jan 31 '13 at 12:16
should work as above, just subclass CachingQuery, so the `__iter__()` here takes the outermost effect, then the `__iter__()` of CachingQuery operates within that. — zzzeek, Jan 31 '13 at 17:37
sorry for the late reply! I decided to stick with the simpler solution as the other didn't work because of various errors and I simply don't have time to try and solve them. Your first solution works so I'll mark that as the correct answer. :) I got a semi related question for you though: How do you go about invalidating cache? Since I don't know which queries will be affected apart from actual table. Is there some way to invalidate all queries on one table or can you get even more specific? Have you thought of this problem as related to the example? Thanks! — moodh, Feb 06 '13 at 07:21

Abstracting away boiler plate sqlalchemy call to allow different options in a try/except block

1 Answers1