0

I am currently using twitter streaming API for my project using tweepy lib and python (2.7.3). The data that I am receiving is at very high rate. Is it possible to slow it down? I read somewhere that the Twitter doesn't like the idea of delaying the streaming for too long.

Another thing that I want to do is to bring in the data, lets say for 5 mins, and after 5 mins data should be sorted out (based upon any thing! e.g. word count of hashtags, etc.). But I think this will slow down my streaming (as I am using one process).

So, can you suggest a way to handle it? mutli-threading? multi-processing?

Hemant
  • 619
  • 2
  • 6
  • 17

1 Answers1

0

You could use a multi thread approach, let's see this problem as a "producer-consumer problem" http://en.wikipedia.org/wiki/Producer-consumer_problem where the producer will be the main thread filling a buffer during 5min and then another thread will consume the buffer (process it it in some way).

Ketouem
  • 3,820
  • 1
  • 19
  • 29