0

I'm trying to write events in a non-blocking fashion so as not to slow down any of our existing processes. It seems like the two options available are:

  • use Twisted's defer object
  • create a python logging handler

Are there other options. Anybody have experience with this?

Background: We're planning to write events to Amazon's CloudWatch service and I'm concerned about latency from doing the PUT request. I'm not actually so concerned about losing a few PUTs if that changes the answer (we're writing the events for alerting purposes and they'll all be purged after a week anyway).

Adam Nelson
  • 7,932
  • 11
  • 44
  • 64
  • 1
    Just add them to a `multiprocessing` `Queue` and have a function reading them out of the `Queue` (blocking if it's empty) and writing them out? – agf Aug 17 '11 at 22:43
  • What? It doesn't seem very coherent to compare Deferreds and logging Handlers as solutions to a single problem. – Jean-Paul Calderone Aug 18 '11 at 03:25
  • Deferreds are obviously much more powerful - but I don't need that power. I just need to log an event to a handler that may take a full second to respond and I can't wait that long. Nonetheless, are Deferreds the easiest solution, or something else? – Adam Nelson Aug 18 '11 at 13:58
  • Deferreds associate results with functions to handle those results. They're not a solution. Maybe you're talking about the asynchronous HTTP client in twisted.web.client? – Jean-Paul Calderone Aug 18 '11 at 20:28

2 Answers2

1

If logging fits your application, and it sounds so, it should be simpler than using twisted. You could log to a thread safe queue in memory and have a separate thread (or several) pull from it and push to the cloud.

That said, twisted may be faster or at least more scalable, but, especially if you don't know it, with a steeper learning curve.

If the threads approach hits limits you can always switch your logging handler to use twisted, without changing your interface to the app, so that's how I would start, with a simple approach and bases covered.

Jürgen Strobel
  • 2,200
  • 18
  • 30
  • I don't think that twisted will be faster or more scalable for this architecture. You have just one client/server connection and twisted is just adding some overhead on top of the logging handling. – schlamar Dec 14 '12 at 10:09
1

This would be trivial with gevent (just do the PUT in a separate greenlet and have socket patched).

tmc
  • 716
  • 7
  • 15