1

Many times I walk into this:

I already have a large ff object(represented by a matrix/array) and then I want to add a new column/row to it, as I have some updated data and don't want to create a new big object from scratch (which can be very time consuming).

I'm trying something like that:

t <- cbind(a = c(1,2,3,4,5), b=c(6,7,8,9,10))
ff.t <- ff(t, dim=dim(t))
# Adding new column works fine
dim(ff.t) <- c(5, 3)

ff (open) double length=15 (15) dim=c(5,3) dimorder=c(1,2)
     [,1] [,2] [,3]
[1,]    1    6    0
[2,]    2    7    0
[3,]    3    8    0
[4,]    4    9    0
[5,]    5   10    0

# Adding new row gives error
dim(ff.t) <- c(6, 4)

dim(ff.t) <- c(6, 4) Error in dim<-.ff(*tmp*, value = c(6, 4)) : you can only change the fastest rotating dim 1: dim<-(*tmp*, value = c(6, 4)) 2: dim<-.ff(*tmp*, value = c(6, 4)) 3: stop("you can only change the fastest rotating dim")

What does this mean ? Is there a way to workaround this and add columns/rows to an ff object (increasing also the filesize naturally) ? If you can't avoid creating a new object, what is the best way to do it ? The new column could be initialized with 0 or NA.

Thanks

Nicolas Meng
  • 21
  • 1
  • Wouldn't it be easier if you converted it to `ffdf`? – LyzandeR Dec 03 '15 at 10:46
  • An ffdf could be an option, though I'm afraid it could be much slower than a traditional ff array representation. My data is numeric only and will have many columns (about 10k), so I guess this would create a file for each column right ? Wouldn't that affect performance ? – Nicolas Meng Dec 04 '15 at 13:39

1 Answers1

0

What I would do is convert your existing object to a data.frame which would make it easy to add columns and rows.

Example of adding a column:

t <- as.data.frame(t)

t["newColName"] <- NA
t$newColName <- *whatever value* (Example: t$a - t$b)
Mick
  • 324
  • 4
  • 14