Write tweets from rtweets package to csv

Question

I'm unable to write tweets from search_tweet() in 'rtweet' package to csv. It throws the following error:

Here's a link to the question I previously asked, that has details on the type of search_tweet() object creates: Class and type of object is different in R. How should I make it consistent?

How should I write this files as csv?

library(rtweet)
comments <- search_tweets(
    queryString, include_rts = FALSE,
    n = 18000, type = "recent",
    retryonratelimit = FALSE)

write_csv(comments, "comments.csv", append =  TRUE)

Error: Error in stream_delim_(df, path, ..., bom = bom, quote_escape = quote_escape) : Don't know how to handle vector of type list.

class(comments)

"tbl_df" "tbl" "data.frame"

screen grab of comments

Seems like `comments` is in a format that `write_csv` does not like. As @sm925 suggested, a reproducible example would help — Chelmy88, Jan 16 '20 at 17:41
`rtweet::search_tweets` clearly says that it returns a `list`. `readr::write_csv` clearly says that it requires a `data.frame`. It seems you are trying to push a square peg through a round hole. I suggest that your question of `write_csv` is wrong; instead you should be asking how to change the returned value from `search_tweets` from a `list` to a `data.frame`. And yes, as always, *actual sample data is important here*. (And at that point, `write_csv` should just work.) — r2evans, Jan 16 '20 at 17:45
The user's previous question indicated that `class(comments)` returned `"tbl_df" "tbl" "data.frame"` so it should be a data.frame but it probably includes a list column. The there is a function `rtweet::write_as_csv` which takes care of the flattening but does not have an `append=` option. — MrFlick, Jan 16 '20 at 17:49
Thanks @MrFlick, while the docs do say `list`, I see that the [source code](https://github.com/cran/rtweet/blob/master/R/search_tweets.R)` disagrees with that (by calling `do.call("rbind", rt)` on the value to be returned). — r2evans, Jan 16 '20 at 17:53
Hi, I have updated the question with relevant details. As it is a load of tweets directly from search_tweets(), I don't know how to show it as a reproducible example. Is there a way I can be more helpful in finding an answer to this question? — Shreya Agarwal, Jan 16 '20 at 17:56
@ShreyaAgarwal using `dput(comments)` in your linked post, I'm able to write data to csv using: `df <- as.data.frame(comments) write.csv(df, "comments.csv")` — sm925, Jan 16 '20 at 18:01
@sm925, it works, but it doesn't allow appending tweets from another handle. I'm running the code above in a for loop. My purpose is to append comments for all the handles in a csv. So, I was keen on adding append to write csv function. — Shreya Agarwal, Jan 16 '20 at 18:04

MrFlick · Answer 1 · 2020-01-16T18:05:51.273

The rtweet package has a function to export to CSV called write_as_csv but for some reason does not expose the append= option. You can take the code of that function and change it to add an append option. For example

write_as_csv2 <- function(x, file_name,
                         prepend_ids = TRUE,
                         na = "",
                         fileEncoding = "UTF-8", append=FALSE) {
  ## to minimize rounding
  op <- options()
  on.exit(options(op))
  options(scipen = 14, digits = 22)

  ## validate inputs
  stopifnot(is.data.frame(x), is.character(file_name), length(file_name) == 1L)
  if (!grepl("\\.csv$", file_name)) {
    file_name <- paste0(file_name, ".csv")
  }
  ## flatten data
  x <- flatten(x)
  if (prepend_ids) {
    x <- prepend_ids(x)
  }
  utils::write.table(x, file_name, row.names = FALSE, na = na,
    fileEncoding = fileEncoding, append=append, sep=",", dec=".", qmethod="double")

  # or
  # readr::write_csv(x, file_name, append =  append)
}
environment(write_as_csv2) <- asNamespace("rtweet")

Then you can call it like

write_as_csv2(comments, "comments.csv", append =  TRUE)

thanks so much for making that attempt. But unfortunately, the function still rejects it: "attempt to set 'append' ignored" — Shreya Agarwal, Jan 16 '20 at 18:02
Didn't realize `write.csv` ignored append. Switched to `write.table`. You could also swap that out for `write_csv` if you prefer. — MrFlick, Jan 16 '20 at 18:06

Write tweets from rtweets package to csv

1 Answers1

Linked