Python: Twitter scraping using "request(POST)" function

Question

I am trying to use twython package to scrap twitter data. Due to the huge number of twitter handles that I am dealing with, I cannot use twitter.lookup_user(user_id=user_batch) command - so I instead tried to use user_dict_batch = tw.request(user_url, method='POST').

Now, according to the twython documentation, the function request is defined as follows:

 request(endpoint, method='GET', params=None, version='1.1')
 Return dict of response received from Twitter’s API

 Parameters:     
 endpoint (string) – (required) Full url or Twitter API endpoint (e.g. search/tweets)
 method (string) – (optional) Method of accessing data, either GET or POST. (default GET)
 params (dict or None) – (optional) Dict of parameters accepted by Twitter API endpoint 
 version (string) – (optional) Twitter API version to access (default 1.1)

 Return type:   
 dict

my question is, what exactly is the "Twitter API endpoint"? For instance, if I am trying to scrap for information about the user whose twitter handle is 3Degrees_Inc, what would be the Twitter API endpoint for this specific user be? I tried to use 'https://twitter.com/3Degrees_Inc' for my value for the parameter endpoint, but it's keep throwing me the Twitter API 403 (forbidden) error....

thanks,

score 0 · Answer 1 · answered Oct 25 '13 at 20:11

First, twython is a wrap-up for RESTfult twitter API, whose docs you can find here. So if you are interested in API users/lookup, its endpoint is users/lookup. Pay attention that this api accepts POST request, since request for that many users may exceed GET limits.

But brief walkthrough twython documentation shows that lookup_user is implemented in EndpointMixin and according to docs implements this same endpoint interface, so you shure can use twitter.lookup_user(user_id=[user1, user2, user3])

def lookup_user(self, **params):
    """Returns fully-hydrated user objects for up to 100 users per request,
    as specified by comma-separated values passed to the user_id and/or screen_name parameters.

    Docs: https://dev.twitter.com/docs/api/1.1/get/users/lookup

    """
    return self.post('users/lookup', params=params)

Python: Twitter scraping using "request(POST)" function

1 Answers1