I'm working on the Twitter API to find the follower list for each user I have in my db. To do this, I have a script with the following parameters:
mongodb_tweets:
url: mongodb://localhost:27017/
database: hatemap
collection: p
mongodb_users:
url: mongodb://localhost:27017/
database: hatemap
collection: p1
twitter:
configuration:
barer_token: AAAAAAAAAAAAAAAAAAAAAAPtPgEAAAAAoVlZ4I0szkcu4dL%2Bhqif%2F%2BF45Oo%3DJbvSo773bskLu1GexDv9Dq1HjuSjfSwfxgLdDXEdlPO5mKyE6G
end_point: https://api.twitter.com/2/users/id/following
What I want to do is change the endpoint every time, more precisely the id within the endpoint which corresponds to the id of the user for which I want to find the list of followers. For example in my db I have users with id "1", "2", "3", I want to change the endpoint within the same execution with all these different ids.
What I want to do is make multiple requests for each endpoint. In the sense that initially I have the following default endpoint: https://api.twitter.com/2/users/id/following, then I modify the endpoint with the replace command like so:
def __connect_to_endpoint(self, retried=False) -> dict:
"""
This method sends the request to twitter and return the response.
The possibles status codes in the twitter response are:
- 200: ok,in this case the response is a valid response;
- 429: rate limit exceeded, this means that either more requests were sent per second than allowed or more requests were sent in 15min than allowed. so in this case this method waits 1 second and tries to send the request again, if twitter still replies with a 429 code, it retrieves from the reply the time when the limit will reset and wait for that time to resubmit the request;
- others: in this case the method raises an exception
:param retried: a parameter that indicate if it is the first retry after an error or not, defaults to False
:type retried: bool, optional
:raise Exception: when twitter response with not 200 or 429 status code.
:return: dict that contains the response from twitter
:rtype: dict
"""
prova = "https://api.twitter.com/2/users/id/following"
to_search = self.mongodb_tweets.get_users_id()
saved = self.mongodb_users.get_users()
to_search = [usr for usr in to_search if usr not in saved]
#self.log.info("LOADED {} USERS ID TO SEARCH".format(len(to_search)))
for id in to_search:
print(id)
self.__twitter_end_point = prova.replace("id", id)
#print(self.__twitter_end_point)
response = requests.request("GET", self.__twitter_end_point, headers=self.__headers, params=self.__query)
if response.status_code == 200:
self.log.info("RECEIVED VALID RESPONSE")
json_response = response.json()
if "errors" in json_response:
self.log.warning("RECEIVED VALID RESPONSE WITH ERRORS")
ids = []
for i in json_response["errors"]:
ids.append(i["value"])
self.log.warning("IMPOSSIBLE TO RETRIEVE THE FOLLOWING USERS:{}".format(ids))
return json_response
if response.status_code == 429 and not retried:
self.log.debug("RETRY")
time.sleep(1)
return self.__connect_to_endpoint(retried=True)
elif response.status_code == 429 and retried:
self.log.info("RATE LIMITS REACHED: WAITING")
now = time.time()
now_date = datetime.fromtimestamp(now, timezone.utc)
reset = float(response.headers.get("x-rate-limit-reset"))
reset_date = datetime.fromtimestamp(reset, timezone.utc)
sec_to_reset = (reset_date - now_date).total_seconds()
for i in tqdm(range(0, math.floor(sec_to_reset) + 1), desc="WAITING FOR (in sec)", leave=True, position=0):
time.sleep(1)
return self.__connect_to_endpoint(retried=True)
else:
self.log.critical("GET BAD RESPONSE FROM TWITTER: {}: {}".format(response.status_code, response.text))
raise Exception(response.status_code, response.text)
This piece of code is the method that allows me to connect to the endpoint. As you can see in the first lines I use the replace command to change the endpoint with the different ids of the users that are present in the database for which I want to find the followers. However, this leads me to find only the followers for the last id but not for all.
