How to extract all tweets from multiple users' timelines using R?

Question

I am working on a project for which I want to extract the timelines of around 500 different twitter users (I am using this for historical analysis, so I'll only need to retrieve them all once- no need to update with incoming tweets).

While I know the Twitter API only allows the last 3,200 tweets to be retrieved, when I use the basic UserTimeline method of the R twitteR package, I only seem to fetch about 20 every time I try (for users with significantly more, recent, tweets). Is this because of rate limiting, or because I am doing something wrong?

Does anyone have tips for doing this most efficiently? I realize it might take a lot of time because of rate limiting, is there a way of automating/iterating this process in R?

I am quite stuck, so thank you very much for any help/tips you may have!

(I have some experience using the Twitter API/twitteR package to extract tweets using a certain hashtag over a couple of days. I have basic Python skills, if it turns out to be easier/quicker to do in Python).

It's probably because of paging. They're not going to send 3200 tweets over the wire at once. I would suggest trying to figure out how the twitter api performs paging and then going from there — Greg, Apr 12 '16 at 19:10

score 0 · Accepted Answer · answered Apr 12 '16 at 19:13

0

It looks like the twitteR documentation suggests using the maxID argument for pagination. So when you get the first batch of results, you could use the minimum ID in that set minus one as the maxID for the next request, until you get no more results back (meaning you've gotten to the beginning of a user's timeline).

answered Apr 12 '16 at 19:13

Matt

646
4
11

Thank you! That's very helpful. – letgolego Apr 19 '16 at 15:12

How to extract all tweets from multiple users' timelines using R?

1 Answers1