I am extracting data from Twitter using Twint. The following code extracts tweets that contain my target keyword. After extracting all the relevant tweets, it changes the data to a dataframe (Tweets_df). While it is running it is storing the data in RAM. This is making it slower. I want to export each tweet and append it to the dataframe when the code is running. any help is appreciated.
c = twint.Config()
c.Lang = "en"
c.Pandas = True
c.Store_csv = True
c.Search = "keyword"
twint.run.Search(c)
Tweets_df = twint.storage.panda.Tweets_df
Here is what I am trying to add: I don't want to wait until it finishes running to export the data.
for index,tweet in Tweets_df.iterrows():
with open('filename', 'a') as csvfile:
f=csv.writer(csvfile)
for i in my_data:
f.writerow(i)
csvfile.close()
