I am trying to download a live stream of tweets using the r package {rtweet}.
Seems when I use hashtags that are not popular I get the following message:
Streaming tweets for 600 seconds...
The stream disconnected prematurely. Reconnecting...
When I use a popular hashtag this does not seem to happen quite so often,
But the stream does eventually disconnect after a while...
I am using a while loop and an if else to append new tweets every 10 minutes to a googlesheet. My code also adds a column with a timestamp for each iteration that runs.
Can anyone help me understand why it seems to work well with popular tweets (i.e. #trump) versus something which isn't trending at the moment? The code is supposed to append an empty row if no tweets with the index hashtag are found during the 10 minutes of streaming. When it works, it works great, but maybe there is something I need to change so that unpopular or non-trending tweets don't cause it to disconnect?
I have read this previous post which seems to indicate a similar problem, but I don't think I should be getting this error with 10 minute intervals. link to previous question I have tried running this script using a popular hashtag and a 3 hour interval and after running overnight it still disconnected prematurely.
In the case that the connection is disconnected, how could I automate re-rerunning the script again in rstudio? Say, if I'm not near my computer when this happens?
Any help much appreciated.
library(rtweet)
library(googlesheets4)
library(googlesheets)
library(googledrive)
googlesheets4::sheets_auth(email = "someemail",
token = "somestring")
ss <- sheets_get("URL to googlesheet goes here")
while (Sys.Date() < "2020-02-15"){
newtweets <- stream_tweets(q = "some hashtags go here",
timeout = 60*10,
file_name = NULL,
parse = TRUE)
if(is.null(newtweets)) {
newtweets <- data.frame(matrix(ncol = 91, nrow= 1))
colnames(newtweets) <- c("user_id", "status_id", "created_at", "screen_name", "text",
"source", "display_text_width", "reply_to_status_id", "reply_to_user_id",
"reply_to_screen_name", "is_quote", "is_retweet", "favorite_count",
"retweet_count", "quote_count", "reply_count", "hashtags", "symbols",
"urls_url", "urls_t.co", "urls_expanded_url", "media_url", "media_t.co",
"media_expanded_url", "media_type", "ext_media_url", "ext_media_t.co",
"ext_media_expanded_url", "ext_media_type", "mentions_user_id",
"mentions_screen_name", "lang", "quoted_status_id", "quoted_text",
"quoted_created_at", "quoted_source", "quoted_favorite_count",
"quoted_retweet_count", "quoted_user_id", "quoted_screen_name",
"quoted_name", "quoted_followers_count", "quoted_friends_count",
"quoted_statuses_count", "quoted_location", "quoted_description",
"quoted_verified", "retweet_status_id", "retweet_text", "retweet_created_at",
"retweet_source", "retweet_favorite_count", "retweet_retweet_count",
"retweet_user_id", "retweet_screen_name", "retweet_name", "retweet_followers_count",
"retweet_friends_count", "retweet_statuses_count", "retweet_location",
"retweet_description", "retweet_verified", "place_url", "place_name",
"place_full_name", "place_type", "country", "country_code", "geo_coords",
"coords_coords", "bbox_coords", "status_url", "name", "location",
"description", "url", "protected", "followers_count", "friends_count",
"listed_count", "statuses_count", "favourites_count", "account_created_at",
"verified", "profile_url", "profile_expanded_url", "account_lang",
"profile_banner_url", "profile_background_url", "profile_image_url", "iteration")
} else {
newtweets[1:dim(newtweets)[1], "iteration"] <- as.character(Sys.time())
newtweets <- apply(newtweets, c(1,2), as.character)
newtweets <- as.data.frame(newtweets)
}
sheets_append(newtweets, ss = ss, sheet = "mytweets")
}
