0

How do I use the list feature in ffloadproperly to selectively load a part of an ffdf?

I have read numerous times that it is possible to selectively load parts of an ff dataframe, but can't seem to get it working myself.

My issue is that I'm trying to perform a merge with a vector that has a list of unique IDs but the ffdf file is too large for my computer to handle this merge.

So far I have been trying to use ffinfo to get a list of the different parts making up my ffdf.

For instance, I get the following list of "file parts" for the ffdf in question:

ffinfo(file=paste(imp_tables_root_loc,table_name,sep=""))

where paste(imp_tables_root_loc,table_name,sep="") specifies the ffdf table and its location

[1] "ffdf1590409e251b.ff" "ffdf159047426082.ff" "ffdf159058a426ab.ff" "ffdf1590410708c.ff"  "ffdf159022b49bf.ff" 
 [6] "ffdf1590174e5dec.ff" "ffdf159045d93226.ff" "ffdf159037f87280.ff" "ffdf159044ad3d39.ff" "ffdf15905224601a.ff"
[11] "ffdf159027936205.ff" "ffdf1590133841c8.ff" "ffdf15902d365cac.ff" "ffdf159065b4259.ff"  "ffdf15904a162908.ff"
[16] "ffdf15905529c1c.ff"  "ffdf1590eda1092.ff"  "ffdf1590ab65eb.ff"   "ffdf159048e74f82.ff"

I then try to ffload by calling:

ffload(file=paste(imp_tables_root_loc,table_name,sep=""),list="ffdf1590409e251b.ff")

But this results in this error:

Error in ffload(file = paste(imp_tables_root_loc, table_name, sep = ""), : not in ffarchive: " ffdf1590409e251b.ff",

bibzzzz
  • 193
  • 1
  • 10

1 Answers1

0

list: An optional vector of names selecting those objects to be restored (default NULL restores all)

So it lets you load only a specific object when you ffsave'd multiple. i.e.

R> a = as.ffdf(data.frame(x = rnorm(10), y = rbinom(10, 5, .2)))
R> b = as.ffdf(data.frame(g = 1:15, h = rpois(15, 5)))
R> ffsave(a, b, file = "foo")
R> rm(list = ls())
R> ffload("foo", list = "a")
R> ls()
[1] "a"
R> rm(list = ls())
R> ffload("foo", list = "b")
R> ls()
[1] "b"

I don't think it allows you to load just one column of an ffdf

Jake Burkhead
  • 6,435
  • 2
  • 21
  • 32