0

I have a large data.table (let's call it DT) in R which takes too long to save as csv (would be great to have a fast writer similar to fread). Therefore I am converting the data.table to a ffdf object:

library(data.table)
library(ffbase)
ffiles <- as.ffdf(DT)

This works and I can see the object. Also when transforming ffiles to a data.frame, by doing

DF <- as.data.frame (ffiles)

.. I see the flat files being opened and loaded. However, if I try to save the ffiles, by doing

save.ffdf(ffiles,dir=path-to-folder, overwrite=TRUE)

I get the error: NA were not saved, because not found

What could cause this?

BTW: I have set fftempdir as options before.

PS: could someone create a "ff" or "ffbase" tag? Don't have enough reputation:-/

user3340145
  • 157
  • 9
  • First, why don't you save the `data.table` using `saveRDS`? – Jan van der Laan Feb 28 '14 at 07:59
  • Second, I can not reproduce the error. I am using the iris dataset. Could you give an reproducible example? Also, `save.ffdf` does not have an `overwrite` argument, so this should also give an error. – Jan van der Laan Feb 28 '14 at 08:02
  • Jan.. thank you for looking into this. I don't have a reproducible example.. it should not happen, but sometimes I get this error and I don't know why... therefore I was wondering what generally could cause this and whether someone else had such an error – user3340145 Mar 01 '14 at 23:25
  • Also, please check again: save.ffdf does have an overwrite argument – user3340145 Mar 01 '14 at 23:27

1 Answers1

0

OK, it seems like one error was based on the way I indicated the path-to-folder. Using "C:\\Folder\\folder\\file" to set the directory seems not to work properly? However, using "C:/Folder/folder/file" seems to do the job.

Nevertheless, I keep on having continuous issues with the ff-files on a network (e.g., they do not load after saving them).

SaveRDS and ReadRDS seem to do a great job for quickly saving and reading in large files (which are also compressed). So thanks for that Jan!

user3340145
  • 157
  • 9