0

I am building a simple application on Google App Engine, which uses twitter ReST Api. I am able to successfully obtain data from a GET request to twitter ReST Api. I am still new to python, so please excuse any ignorance. So my question is, what is the most efficient way to make my GET request to Twitter, so I can get updated information? Someone suggested using timer() and make the http request every 5 seconds but that doesn't seem efficient, does it? The idea is, i want to keep my result set updated and the only way to do that is to keep making calls to the api. I did some research online and found google.appengine.api.urlfetch, however because i am new to this, i do not know if it would be the correct thing to do. any advice?

  • 1
    I know what you're trying to do, and simply don't think its possible. You're looking for live notifications from Twitter, right? There needs to be some system to check whether the state on Twitter has changed, which is why people suggest using a timer. As you may have noticed, this is pretty inefficient as you tend to make a lot of requests. However, how else could you possibly know if something has changed without checking? There are methods for writing live notifications that are more efficient, but I don't think they would work here. – Mmm Donuts Dec 27 '15 at 17:44
  • thanks for your response. I appreciate it. It is true, a timer would be very inefficient. –  Dec 27 '15 at 19:03
  • No problem. I could tell you how facebook does live notifications if you're interested. Basically a request with a long timeout is made to the server from the client. The request, since it doesn't timeout, sits there in limbo waiting for a response from the server. However, the server only responds when the state has changed, closing the timeout and thus, a live update is sent to the client. Another request is made the same way and the process is repeated. I'm not sure if you could make use of that, but that's one way to get efficient, live updates. – Mmm Donuts Dec 27 '15 at 19:13
  • Unless you can find a feed api (which would require a long running process and therfore a managed vm) you have no alternative but to poll over a period. This would then be done using a combination of cron/tasks. – Tim Hoffman Dec 28 '15 at 02:10
  • @TimHoffman, "a long-running process" does *not* require a managed VM -- you can perfectly well use a manual-scaling module in classic App Engine for that. If, that is, you CAN find a feed API for Twitter, a completely different issue -- just correcting the "long-running process" issue. – Alex Martelli Dec 28 '15 at 23:07
  • @AlexMartelli yeah I had forgotten about the "manual scaling" haven't used it yet. – Tim Hoffman Dec 29 '15 at 05:28
  • @TimHoffman, I use manual scaling (in some secondary module) when I need to perform (with but modest amounts of scaling) some background task that can't be guaranteed to end in 10 minutes and would be a bother to checkpoint-and-restart (say, on a task queue) -- the lack of deadline can often make the coding SO much easier! – Alex Martelli Dec 29 '15 at 18:05

1 Answers1

0

You're mixing up a few things. The urlfetch api is just a way of making requests from GAE, unrelated to polling itself.

If you can tolerate polling rates with a resolution of 1 minute you can use the cron service to drive your polling.

You can get resolution under 1 minute with the deferred queue service, which is more efficient than using timers.

Update:

The Task Queues are preferable to the deferred library, the deferred functionality is available using the optional countdown or eta arguments to taskqueue.add():

  • countdown -- Time in seconds into the future that this task should run or be leased. Defaults to zero. Do not specify this argument if you specified an eta.

  • eta -- A datetime.datetime that specifies the absolute earliest time at which the task should run. You cannot specify this argument if the countdown argument is specified. This argument can be time zone-aware or time zone-naive, or set to a time in the past. If the argument is set to None, the default value is now. For pull tasks, no worker can lease the task before the time indicated by the eta argument.

Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97