Twitter4j - Rate limit exceeded

Question

I would like to get followers using getFollowersIds() in Twitter4j, but I get

ConnectionErrorException ... rate limit exceeded

public static void main(String[] args) {
        try {
            Twitter twitter = TwitterFactory.getSingleton();
            String[] srch = new String[]{"TechCrunch"};
            ResponseList<User> users = twitter.lookupUsers(srch);
            for (User user : users) {

                UserHarvest us = new UserHarvest(6017542);
                us.getFollowersIds();
                try {
                    us.getContributors();
                } catch (ConnectionErrorException ex) {
                    Logger.getLogger(UserHarvest.class.getName()).log(Level.SEVERE, null, ex);
                }
            }
        } catch (TwitterException ex) {
            Logger.getLogger(UserHarvest.class.getName()).log(Level.SEVERE, null, ex);
        }
    }

Error message:

Exception in thread "main" harvest.twitterharvest.ConnectionErrorException: Connection could not have been established
    at harvest.twitterharvest.WrapperTwitter4J.getFollowersIDs(WrapperTwitter4J.java:75)
    at harvest.twitterharvest.UserHarvest.getFollowersIds(UserHarvest.java:106)
    at harvest.twitterharvest.UserHarvest.main(UserHarvest.java:140)
Caused by: 429:Returned in API v1.1 when a request cannot be served due to the application's rate limit having been exhausted for the resource. See Rate Limiting in API v1.1.(https://dev.twitter.com/docs/rate-limiting/1.1)
message - Rate limit exceeded
code - 88

Relevant discussions can be found on the Internet at:
    http://www.google.co.jp/search?q=92c30ec6 or
    http://www.google.co.jp/search?q=19e1da11
TwitterException{exceptionCode=[92c30ec6-19e1da11], statusCode=429, message=Rate limit exceeded, code=88, retryAfter=-1, rateLimitStatus=RateLimitStatusJSONImpl{remaining=0, limit=15, resetTimeInSeconds=1384512813, secondsUntilReset=904}, version=3.0.4}
    at twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:162)
    at twitter4j.internal.http.HttpClientWrapper.request(HttpClientWrapper.java:61)
    at twitter4j.internal.http.HttpClientWrapper.get(HttpClientWrapper.java:89)
    at twitter4j.TwitterImpl.get(TwitterImpl.java:1894)
    at twitter4j.TwitterImpl.getFollowersIDs(TwitterImpl.java:409)
    at harvest.twitterharvest.WrapperTwitter4J.getFollowersIDs(WrapperTwitter4J.java:73)
    ... 2 more

I see secondsUntilReset=904. I run code 1 hour later and get the same error message. I don't know why. Thanks for any help.

This is a twitter problem, not a Twitter4J problem. Twitter4J just calls into the Twitter API. Has the twitter account your using been blacklisted? Has it been banned? Maybe Twitter have stopped providing you with data for too many calls? — Ben Dale, Nov 15 '13 at 13:54
The account is not blacklisted or banned. Is it possible twitter have stopped provide me data for more than one day? — Matt, Nov 15 '13 at 16:10

score 10 · Accepted Answer · answered Nov 15 '13 at 15:57

10

There are rate limits in the Twitter API. You cannot call a given Twitter API endpoint more than a given number of times per 15 minutes (on behalf of the authencated user or not).

What happened to you is that your code must have reached the rate limit very quickly (the endpoint to retrieve followers IDs is limited to 15 calls per 15 minutes for a given authenticated user) so you will have to wait (904 seconds) before trying again.

Be careful to the Twitter API endpoints you are calling (through Twitter4J) in order to economize your API calls and thus avoid reaching the rate limit.

For further informations, have a look at those resources on Twitter Developers, especially at its documentation :

REST API Rate Limiting in v1.1 (general topic about rate limiting in the Twitter API)
REST API v1.1 Limits per window by resource (rate limits for each endpoint)
Error Codes and Responses (which explains to you what is your '88' response code).

answered Nov 15 '13 at 15:57

air-dex

4,130
2
25
25

I know there are limits in the Twitter API, but I don't understand why I get the same error message one hour later and in the meanwhile I haven't used Twitter API services. – Matt Nov 15 '13 at 16:20
1

How many users do you have in your `users` list? As I said in my answer, you can only make 15 requests per 15 min to retrieve followers IDs. So if your `users` list has got at least 16 users (15 + 1), the 16th member will make a 16th request in less than 15 minutes to the `GET followers/ids` endpoint. The Twitter API will consequently respond 429 and the exception will be raised. This scenario will happen each time to will try to execute your code (if the `users` list has got > 15 members, of course), even one hour after the previous execution. – air-dex Nov 16 '13 at 14:59
I have only one user in list. As you can see "TechCrunch" in example – Matt Nov 17 '13 at 12:31
1

I don't think that you understand what your code **really** does. First it searches "`TechCrunch`" users on Twitter. Then it retrieves the followers IDs and the contributors of each and every one of them. And it looks like that your reseach returns you far more than one single user. I did the research on the Web interface ( https://twitter.com/search?q=TechCrunch&src=typd&mode=users ) and it returns me more users to reach the rate limit than required: @TechCrunch (of course!), @TCFR, @TechCrunchOnion, @TechCrunchTV, @RSS_TechCrunch, @TechCrunchItaly, @TechCrunchIndia, @TC_Europe, @alexia... – air-dex Nov 17 '13 at 15:10
These links has changed now, here are the updated ones: rate limits: https://developer.twitter.com/en/docs/basics/rate-limits.html, response codes: https://developer.twitter.com/en/docs/basics/response-codes – A. Harkous Jun 17 '18 at 12:39

score 3 · Answer 2 · answered Nov 19 '13 at 04:46

3

You dont have to wait to fix rate limit issue. If you have many tokens corresponding different twitter accounts, then you can renew your current used token, right. Think about the big picture. Let's say that you have 25 different accounts(tokens) in a list or array object, and you are getting one of them each time that you have a rate limit issue. Then when you reach to the last item in list token list, then you can refill this list with these 25 token accounts, right. The beauty is this, if you have good enough number of accounts(tokens), each refilling loop, the system automatically will be waited to recover the used accounts, so you will not have to worry about the rate limit and your application will never stop because of rate limit issue. I have already implemented this mechanism in my project and it works perfect. This is the only best solution.

answered Nov 19 '13 at 04:46

mgokhanbakal

1,679
1
20
26

1

Hey man, how would you check for a 429 TwitterException during a loop to then change the tokens of the twitter object? – nullwriter Mar 16 '14 at 13:49
Actually, I was checking the error message whether it contains "Rate limit exceeded" or not. If it contains, I renew the current token. It was the only way at that time that I used. Maybe, now twitter4J has a machanism for that. I dont know. – mgokhanbakal Mar 16 '14 at 15:39
@MehmetGokhanBAKAL are you still here? I have a question. – user3764893 Jun 27 '14 at 14:40
Sure, You can ask your question. – mgokhanbakal Jun 28 '14 at 09:13
The solution you have mentioned above would make sense for a single user, using that app. But what if there are 10.000 user using that app at the same time? With only 25 accounts(tokens) that would be unhandleable, right? – user3764893 Jun 28 '14 at 13:12
Yes, that is correct. However, my application is not a public application. What I mean is I am the only user for the application just for academic purposes. :D If you wanted to develop a publicly available application which would be used by many users, it would be unhandleable. :D – mgokhanbakal Jun 29 '14 at 10:18
@MehmetGokhanBAKAL How did you create 25 different tokens? I am assuming it means 25 different accounts. To do that every new account needs a different mobile number. If you use the same mobile number twitter says "The phone number you're trying to use is already associated with another Twitter account.". I don't know how you have achieved this. Please explain. – Ravindra S Aug 02 '15 at 16:44
The old policy was different. When I created more than one account, there was no obligation like providing a valid phone number to create an account. So, that is why I was able to create many accounts and tokens. – mgokhanbakal Aug 03 '15 at 14:17

Twitter4j - Rate limit exceeded

2 Answers2

Linked