0

I was developing a Push System with Tornado. As i'm taking long polling, i need to keep a list of Future Object in order to set results for them later. Then i want to keep the Future list in Redis, so i 'dumps' each Future with Pickle module and set it to Redis, But after i get it from Redis, and 'loads' it. I found it was not the original Future Object, and when i invoke set_result function on this future object, it doesn't work as my expectation.

Any one can help me on this?

This is the part of my code:

@singleton
class MessageProxy:

    def register_subscriber(self, subscriber):
        r.set("subscriber", pickle.dumps(subscriber))

    def send_message(self, message):
        subscriber = pickle.loads(r.get("subscriber"))
        subscriber.set_result(message.content)
laike9m
  • 18,344
  • 20
  • 107
  • 140
yunfeng.guo
  • 175
  • 1
  • 11

1 Answers1

0

Future objects cannot usefully be pickled. An unfinished Future is just a placeholder with a list of callbacks, and the callbacks cannot be pickled. (furthermore, the list of callbacks can be changed at any time; the fact that you're able to call pickle.dumps() without an exception means you're doing it too early; the application hasn't had a chance to attach its callback yet).

Instead of storing Futures in redis, you'll have to keep a map in memory and store identifiers in redis. If you have multiple server processes, then of course you'll have multiple of these maps. You'll have to ensure that messages get where they need to go either by broadcasting every message to every server (without knowing who is listening to what) or by including some sort of addressing information in redis.

Ben Darnell
  • 21,844
  • 3
  • 29
  • 50
  • Thanks Ben, I stopped pickling Future and made 'maps' for each instance( process ), but finally i found i still can not control all Future from different instances once i request to send message to all of those. I think the ' share Future' solution is not right in my Push System. – yunfeng.guo Mar 26 '15 at 09:43
  • And i know Redis Pub/Sub can be used, i found the related Async library for tornado. So now i'm so curious about how it can work under multiple processes environment, how does it control the IOLoop or Object from different instances? – yunfeng.guo Mar 26 '15 at 09:46
  • You don't "control the IOLoop from different instances". The idea behind pubsub is that you publish events without knowing who (if anyone) is listening. One node leaves messages in redis; another node (or nodes) picks them up. – Ben Darnell Mar 26 '15 at 12:26
  • Ok, i got it. i will dive into redis sub/pub later. Do you think it's possible to use Future to implement this, or any other solution without Redis Pub/Sub, i mean i don't really want to touch Redis Pub/Sub before i exactly know its mechanism, it's like a magic box for me currently. Any other 'pure tornado' solution can do this? – yunfeng.guo Mar 27 '15 at 02:21