Stack Overflow
Stack Overflow

Questions tagged [ffbase]

Basic statistical functions for package ff

R package ffbase provides basic functionality to allow to do basic data operations on ff structures. This functionality is also available in the R base library and ffbase tries to provide the methods needed for ff structures such that switching between in-RAM objects and on-disk objects in ff is more straightforward.

More info:

64 questions
2
votes
1 answer

Replace NAs in a ffdf object

I`m working with a ffdf object which has NAs in some of the columns. The NAs are the result of a left outer merge using merge.ffdf.I would like to replace the NAs with 0s but not managing to do it. Here is the code I am running: library(ffbase) …
ddg
  • 2,493
  • 2
  • 20
  • 23
1
vote
0 answers

When dealing with big excel in R, java.lang.OutOfMemoryError, how to save big excel with ffbase?

I was stuck at the last step, saving my excel. The error is : Error in .jcall(cell, "V", "setCellValue", value) : java.lang.OutOfMemoryError: GC overhead limit exceeded therefore I searched and tried: > options(java.parameters=’-Xmx4g’) Error:…
kai
  • 23
  • 5
1
vote
0 answers

could not find function "as.data.frame.ffdf"

I'm following the guide Big Data Analytics with R. But the as.data.frame.ffdf function seems to be missing. Does anyone have an idea? Or is there any alternative solution? Here's sample code: # Data were downloaded from Bureau of Transportation…
Richard Bai
  • 11
  • 1
1
vote
1 answer

ff: returning multiple arrays with a single ffapply function call

I am dealing with a large dataset of 3D imaging data that I have loaded in to R using ff(). require(ff) nSubj <- 125 vol_dim <- c(139,137,87) ff_qmap <- ff(0, dim=c(vol_dim,nSubj) Simple calls like getting an average array/"volume" back work…
user10023347
  • 141
  • 1
  • 5
1
vote
1 answer

One-to-many using ffbase in R

I would like to replicate the following one-to-many join using ffdf. What would be the best way to do this? Below I present an example of what I would like to get, using data.tables. I am aware of the following description of the merge.ffdf…
dleal
  • 2,244
  • 6
  • 27
  • 49
1
vote
1 answer

Reassigning values to columns in ffdf [R]

I am having trouble doing the following operations in a larger dataset. I wonder if there is a built in way to do it with either ff or ffdf. Example: Modifying a character columns in an ffdf object using substr and reassign it as a different…
dleal
  • 2,244
  • 6
  • 27
  • 49
1
vote
0 answers

R ffdfdply appending error

I want to cast my data (data.frame) from long to wide format with the values of "ITEM" as columns and values ("ITEM2") (see below): Long format Wide format Therefore I use the dcast-function from the package reshape2: df <= dcast(df,SEQUENCEID +…
silem
  • 21
  • 7
1
vote
1 answer

Splitting an ffdf object

I'm using ff and ffbase libraries to manage a big csv file (~40Go and 275e6 observations). I'd like to split/partition this file according to one of its columns (which is a factor column). With a normal data frame, I would do something like that: a…
G. Lombardo
  • 90
  • 7
1
vote
1 answer

log2 transform ff objects

I would like to log2 transform all numeric values in a ff object from ffpackage. Using my df: library(ff) df <- 'probeset_id sample1 sample2 sample3 probe_1 1834.2 1743.4 1384 probe_2 4711 4922 4650 probe_3…
user2120870
  • 869
  • 4
  • 16
1
vote
1 answer

Adding a column with character data to a ffdf

I've tried to add a Source column to my ffdf, but can't seem to get it to work... if it was a normal df I would simply write mtcars$NewCol <- "AB" If I do this for the ffdf it returns an error require(ff) require(ffbase) mtcarsff <-…
Jacob Odom
  • 216
  • 1
  • 8
1
vote
1 answer

Why summarise in ffbase2 (dplyr_ffbase) shows "error in as.vmode.default() (list) object cannot be coerced to type 'double'"?

I have a large (23 Mln rows) ffdf table (tbl_ffdf) with 10 columns, 7 of them are factors and 3 contain numbers. It looks something like this: TABLE_bad F1 F2 F3 F4 F5 F6 F7 N1 N2 N3 1111 01.15 05.14 busns…
inscaven
  • 2,514
  • 19
  • 29
1
vote
0 answers

How to delete (or select) specified rows or columns of ff matrix, or to subset ff matrix?

A ff matrix of 300,000 rows and 1000 columns: x <- ff(1: 100000000, vmode = "integer", dim = c (300000, 1000), dimorder = c (2,1)) I want to delete the last line of the matrix use the command: x[-300000,] However,I got the error: "can not allocate…
user4530904
  • 11
  • 2
1
vote
2 answers

duplicated function fails for ff date vectors

Hi I am trying to remove duplicates from a ff vector that contains dates using the duplicated function of the ffbase package and the following code: v1 <- c("24-Mar-94", "24-Mar-94", "27-Mar-94", "28-Jun-1986", "29-Jul-1988", "28-Jun-1986",…
NickD1
  • 393
  • 1
  • 4
  • 14
1
vote
1 answer

Reshape ffdf dataframe in R

I am using dcast function to rshape datframe in R, but while using large dataframe. I converted that into ffdf dataframe unable to use dcast function please help me if any alternatives. Find the below example i used for small dataframe and what i…
Naga Pavan
  • 11
  • 3
1
vote
1 answer

Read data from multiple CSV files into single ffdf object

Is it possíble to load at once data from several files into a ff data frame (ffdf)? Lets say I have big_file_part1.csv big_file_part2.csv big_file_part3.csv I know I could load each csv file to a separate ffdf object and then ffdfrbind.fill them…
LucasMation
  • 2,408
  • 2
  • 22
  • 45
1
2
3 4 5