0

I text mined twitter data with the rtweet package, but after saving it to a data frame I am not able to use packages such as ggplot2 to their full potential. What am I doing wrong?

I text mined some twitter data using the rtweet library, and did so in parts so I do not exceed by API limit. After collecting all the data I needed I combined it all into one data frame. I downloaded the dplyr and ggplot2 packages and wanted to visualise tweets over time, but time time variable is not recognised when it comes from the data frame. But if I use one of the batches of mined data with its original name, it is recognised as a time variable and plotted nicely. This is the code I used to mine the data, save it to data frames and combine them all into one data frame.

##data splits used for mining - 7/8 companies at a time 
#1st batch
food_comp1 <- get_timelines(c("accentcatering","angussoftfruits","wmbarrowcliffes","brewdog","brewhouse","capestonefarm","stpierregroupe"), n=3200)
#2nd batch
food_comp2 <- get_timelines(c("cavedirect","brockmoor","cherryfieldltd","fazendagroup","ddcfoods","dairypartners","drakeandmorgan"), n=3200)
#3rd batch
food_comp3 <- get_timelines(c("drinkwarehouse","etsteas","fentimansltd"), n=3200)
#4th batch
food_comp4 <- get_timelines(c("goustocooking","edinburgh_gin","innisandgunn","grapetreefoods","kkfinefoods","finnebrogue","thewhiskyshop"), n=3200)
#5th batch
food_comp5 <- get_timelines(c("onarollsandwich","potsandco","primacheese","purecircle","silburyoils","thealchemistuk"), n=3200)
#6th batch
food_comp6 <- get_timelines(c("artisan_glasgow","thebigprawnco","foodfellas","freshfoodco","wagyurestaurant","charlesfaram","wenzelsthebaker","westerrosssalmo"), n=3200)
#7th batch
food_comp7 <- get_timelines(c("elitefoods_","fevertreemixers","gordon_macphail","gosh_freefrom","specialitydrink"), n=3200)

#merging the datasets into 1
comp <- rbind(food_comp1,food_comp2,food_comp3,food_comp4,food_comp5,food_comp6,food_comp7)
write_as_csv(comp, file_name = "comp", prepend_ids = TRUE, na="", fileEncoding = "UTF-8")
comp <- read.csv("comp.csv", header = TRUE)
View(comp)
#creating a subset with the variables I need 
compsubset <- subset(comp, select=c("created_at","screen_name","text","display_text_width","favorite_count","retweet_count","hashtags","media_type","lang"))
write_as_csv(compsubset, file_name = "compsubset", prepend_ids = TRUE, na="", fileEncoding = "UTF-8")

##final dataset with 9 variables
compsubset <- read.csv("compsubset.csv", header=TRUE)

#attemting to create a ggplot with created_at as the time variable (displays year-date-time)
 ggplot(data = compsubset, aes(x = created_at)) +
     geom_histogram(aes(fill = ..count..)) +
     theme(legend.position = "none") +
     xlab("Time") + ylab("Number of tweets") + 
     scale_fill_gradient(low = "midnightblue", high = "aquamarine4")

Error: StatBin requires a continuous x variable: the x variable is discrete. Perhaps you want stat="count"?

The same code with one of the mined batches before converted to a data frame

ggplot(data = food_comp1, aes(x = created_at)) +
     geom_histogram(aes(fill = ..count..)) +
     theme(legend.position = "none") +
     xlab("Time") + ylab("Number of tweets") + 
     scale_fill_gradient(low = "midnightblue", high = "aquamarine4")

After this last ggplot I got a nice bar chart with tweets over time

Similar thing happens when I try to use dplyr to filter tweets by time, but works fine with the mined data before putting into dataframe

 comptrial %>%
     dplyr::filter(created_at >= "2018-01-01") %>%
     dplyr::filter(created_at < "2019-01-01") %>%
     dplyr::group_by(screen_name) %>% 
     ts_plot("days", trim = 1L) +
     ggplot2::geom_point() +
     ggplot2::theme_minimal() +
     ggplot2::theme(
         legend.title = ggplot2::element_blank(),
         legend.position = "bottom",
         plot.title = ggplot2::element_text(face = "bold")) +
     ggplot2::labs(
         x = NULL, y = NULL,
         title = "Frequency of Twitter statuses posted by SMEs",
         subtitle = "Twitter status (tweet) counts
        caption = "\nSource: Data collected from SME's REST API via rtweet"
     )

Error in seq.POSIXt(data[[dtvar]][1], data[[dtvar]][length(data[[dtvar]])], : 'to' must be of length 1 In addition: Warning messages: 1: In Ops.factor(created_at, "2018-01-01") : ‘>=’ not meaningful for factors 2: Factor screen_name contains implicit NA, consider using forcats::fct_explicit_na 3: Factor screen_name contains implicit NA, consider using forcats::fct_explicit_na 4: Factor screen_name contains implicit NA, consider using forcats::fct_explicit_na

NelsonGon
  • 13,015
  • 7
  • 27
  • 57
  • In the future, please correctly format and punctuate your post for easier reading. Might also help if you made it as concise but reproducible as possible. – NelsonGon Aug 18 '19 at 15:53
  • Do you have `NA`s in your `compsubset` data? Can you try `ggplot(data = na.omit(compsubset), aes(x = created_at)) + geom_histogram(aes(fill = ..count..)) + theme(legend.position = "none") + xlab("Time") + ylab("Number of tweets") + scale_fill_gradient(low = "midnightblue", high = "aquamarine4")` – Ronak Shah Aug 19 '19 at 05:28
  • There are NAs in the other columns, but the one I am using is with mined tweets, where there are no NAs. I will try what you suggested, hope it works better than what I've done so far. – Silvia Ivanova Aug 23 '19 at 15:22

0 Answers0