0

i am trying to load a csv file containing : 517 000 line, and only 20 variables, i am using read.table.ffdf, and it gives the error :

Error in ff(initdata = initdata, length = length, levels = levels, ordered = ordered,  : 
  write error

i used the line :

dépenses<- read.table.ffdf(
  file="COVID-19_Expenditures_Report.csv", 
  sep=",", 
  VERBOSE=FALSE, 
  header=TRUE, 
  colClasses=NA
)

i have seen similar questions with the same error due to the high number of variables, but mine isn't due to that, as i have previously used the same function to read more variables than 20, successfully.

user438383
  • 5,716
  • 8
  • 28
  • 43
math geek
  • 1
  • 1

1 Answers1

0

Try data.table with fread, it loads in your data very fast without any issues.

library(data.table)
depenses <- fread(file="COVID-19_Expenditures_Report.csv", sep=",",  header = T)
Merijn van Tilborg
  • 5,452
  • 1
  • 7
  • 22
  • i have read it using other functions, but for teaching purpposes i need to use ff and it doesn't seem to work, i have also discovered that ff isn't loading any of my previous loaded files, it seems like i have changed something in the settings, but i cann't figure out what it is – math geek Nov 01 '21 at 11:11
  • add the fill = T to your read.table.ffdf() and it works, however still throws a EOF warning. It is insanely slow though using read.table.ffdf taking 30+ seconds, while with fread you get the data within a second or 2-3. IMO especially for teaching purposes it might be important to take speed into account. Rather frustrating if they want to bring it in practice with a 10 fold data set someday... – Merijn van Tilborg Nov 01 '21 at 12:04