How to use col_types using readr's read_delim_chunked?

Question

I am trying to read a file in chunks and specify the col_types, see MWE

write.csv(cars, "cars.csv")


library(readr)
readr::read_delim_chunked("cars.csv", function(x, i) {
  x
}, delim= ",", col_types = cols(
  speed = col_character()
), chunk_size = 10)

but I get erroneous output

NULL

but the non-chunked version works fine

library(readr)
readr::read_delim("cars.csv", delim= ",", col_types = cols(
  speed = col_character()
))

That's a warning, not an error: the code still runs, right? – camille Sep 05 '19 at 03:50 — camille, Sep 05 '19 at 03:50
@camille no. it's an erroneous output. – xiaodai Sep 05 '19 at 04:55 — xiaodai, Sep 05 '19 at 04:55

score 1 · Accepted Answer · answered Sep 05 '19 at 03:32

1

The issue would be that when we do the write.csv, the row.names were included as a new column

write.csv(cars, "cars.csv", row.names = FALSE, quote = FALSE)

Also, we need col_character() instead of col_character

readr::read_delim_chunked("cars.csv",  DataFrameCallback$new(function(x, i) {
  x
}), col_types = cols(
  speed = col_character()
), delim= ",",  chunk_size = 10)

answered Sep 05 '19 at 03:32

akrun

874,273
37
540
662

1

I think the issue is `DataFrameCallback$new` – xiaodai Sep 05 '19 at 04:55
Sorry the () is a typo – xiaodai Sep 05 '19 at 04:57

score 0 · Answer 2 · answered Sep 05 '19 at 04:56

For some reason, you need to wrap the function in DataFrameCallback$new for reasons that I don't understand.

write.csv(cars, "cars.csv")

Works

readr::read_delim_chunked("cars.csv",  DataFrameCallback$new(function(x, i) {
  x
}), col_types = cols(
  speed = col_character()
), delim= ",",  chunk_size = 10)

Gives error

readr::read_delim_chunked("cars.csv",  function(x, i) {
  x
}, col_types = cols(
  speed = col_character()
), delim= ",",  chunk_size = 10)

How to use col_types using readr's read_delim_chunked?

2 Answers2