0

I have a question about a function attribut from the ff package. I have an ff object in a specific directory I want to load. This is what I am using:

setwd(~/Documents/Data)
ffload("Object", overwrite=TRUE)

With OVERWRITE=TRUE, a copy of the object will be saved in a tmp folder and the object will be load from this directory:

[1] "tmp/Rtmpsj1b2f/clone11876c2c0949.ff"

But if you don't want a copy of your object you need to put FALSE to overwrite. If it is the first time you load your ff object it will create a copy in a tmp folder even if overwrite=FALSE! Then I deleted the copy, and re-loaded the object for the second time, I had the following message and no copy:

Warning :
In FUN("Obs_zone_212_19700801_19750731"[[1L]], ...) :
did not overwrite object 'Obs_zone_212_19700801_19750731'

The problem is that I am working with a lot of big files and I don't want to create copies! What can I do ?

Chika
  • 1,455
  • 2
  • 16
  • 24

1 Answers1

0

Maybe it is good to give an explanation about ffload for your case

You have 2 files in your ~/Documents/Data directory. One called Object and another one called Object.ffData. Object.ffData contains the ff files with the raw data. Object contains an R workspace with the virtual attributes of the ff files (like levels e.g., pointers to the ff files, ...) which is only a few Kb depending on your data structure.

ffload unzips the ff files which are contained in Object.ffData into the folder where you have created the ff files (see getOption("fftempdir") - for your case this was in the tmp/Rtmpsj1b2f folder. If you specify the rootpath argument to ffload, you can unzip the ff files to another directory. As in ffload("Object", overwrite=TRUE, rootpath = "/home/myname/myfolder"). The overwrite argument of ffload indicates that if that ff file which you are unzipping already existed in that directory, it will overwrite it. So far the ff part.

When loading with ffload your file, you are also loading the virtual part in your R session. This is for your case an object which you have named Obs_zone_212_19700801_19750731 (see ls()). The warning you see is that when ffloading your new data, it did not overwrite this object. So you need to rm(Obs_zone_212_19700801_19750731) before ffloading to get rid of the warning.

Hope this helps. If your data are ffdf objects, you can also look at save.ffdf from package ffbase. This does not zip your data but keeps it straight on disk.

  • Thank you for this answer! So in fact my problem comes from ffsave. Before loading the ff objects, I created them with ffsave. When I use ffsave, it will create Object.RData and Object.ffdata in the directory I choose and another folder in tmp (the Rtmpsj1b2f folder). If I don't want to have this last folder I need to add the roothpath I want in ffsave? – Chika Jan 21 '14 at 14:53
  • You should set the fftempdir folder before creating the ff objects to a folder where you want your ff files to be located. options(fftempdir = "path/to/your/folder") –  Jan 21 '14 at 15:01
  • Ok! When I add that, I have en error: cannot open file '/tmp/RtmpBHIZfb/file1a8256d52308' : No such file. It seems that even if I asked for a specific folder it doesn't work – Chika Jan 21 '14 at 15:20
  • You need to set fftempdir **before** creating the ff objects –  Jan 21 '14 at 15:24