Stack Overflow
Stack Overflow

Questions tagged [ff]

An R package that provides memory-efficient storage of large data on disk and fast access functions

The ff package provides data structures that are stored on disk but behave (almost) as if they were in RAM by transparently mapping only a section (pagesize) in main memory.

More information:

165 questions
0
votes
1 answer

Assign value to variable in ff

I am working with a ff dataset (final_faf_data) because of the size of my database. I am trying to create a new variable dpy which has only only one value (365) for all rows. When I try the following cmd (similar to what I would do in a…
Krishnan
  • 1,265
  • 2
  • 13
  • 24
0
votes
0 answers

How can I assign data to dynamically named ff variables in R?

I want to read large arrays from .mat files into dynamically named ff files in R. I tried the following: for(i in 1:numFiles){ varName = paste0("ffInter",i) # name of the new ff variable temp = readMat(Files[i]) #reads the array from .mat…
tholor
  • 55
  • 1
  • 6
0
votes
0 answers

Issue with ETLUtils and ODCB connection

I am attempting to connect to an Oracle database, extract my data and put it into a flat file on R. The table has ~ 16 million rows with 25 columns. There are many strings in some of these columns. I am using the RODBC and ETLUtils package in…
SniperBro2000
  • 113
  • 1
  • 5
0
votes
0 answers

reading txt files using ff library in R

I have a large dataset (1.1GB) in tab seperated format. When I read this dataset into the program R using the normal read.table function: data <- read.table(file="C:/Localdata/Postcode model/Data/FinalDataset.txt", header=TRUE,…
Emrys Komen
  • 63
  • 2
  • 10
0
votes
1 answer

How can you apply a function or logical test to a ffdf?

This is basically asking how to apply a previous question to a ffdf : R - applying ifelse to a whole data frame I am basically trying to do the same thing. Given a ffdf dataframe, I am basically asking how to apply an ifelse to the entire dataframe.…
Brian Jackson
  • 409
  • 1
  • 5
  • 16
0
votes
1 answer

Getting list of column classes for all ffdf columns

I have a ffdf with several columns. I would like to find the column classes that read.table.ffdf automatically determined from the data as a vector of strings. A crude way of doing this seems to be looping over the columns and using…
user3817455
  • 51
  • 5
0
votes
0 answers

Memory-related error: ff package on Windows 2008 server

I am trying to do the simplest indexing of an ff vector in R v3.1.0, on a Windows 2008 server. The following basic code: x = ff(1:10) x[ff(2:3)] Returns the error: Error in ffindexget(x, i) : cannot allocate memory block of size 67108864 Tb In…
Audrey
  • 212
  • 4
  • 15
0
votes
1 answer

Subsetting ffdf in loop

I'm trying to subset a very large ffdf object in a loop using ffbase, but I'm getting the error message: Error in UseMethod("as.hi") : no applicable method for 'as.hi' applied to an object of class "NULL" I'm running this code on an ssh with large…
ostionalzax
  • 25
  • 3
0
votes
1 answer

How to change an ff vector to a normal vector R

I have a function that accepts vectors as inputs. I have an ffdf named X, and would like to use columns of data as vectors for the function inputs. To take a column of data named "Mag" as a vector I would use the following: X[['Mag']] However this…
user3565975
  • 75
  • 2
  • 9
0
votes
0 answers

R Foreach parallel processing with ffdf mapply function

I have a large ffdf named 'Scenarios' that I am applying a function to from the NGA package. I am already using mychunks to try and speed things up but it is still slow. Could I run it with parallel processing as well using say the Foreach package?…
user3565975
  • 75
  • 2
  • 9
0
votes
0 answers

Writing a Loop using ffwhich function from ff package R

I have a very large ffdf named 'Scenarios' and I would like to sort one of the columns named 'Eventfreq' into bins based on two variables and then take the sum of the values within each BIN. The function below essentially does this for one set of…
user3565975
  • 75
  • 2
  • 9
0
votes
0 answers

Creating dummy variables in a FF object

I have an FFDF object that I'd like to create dummy variables in. Specifically, I have a separate vector of names and I want to make dummy variables if a name in that vector matches a name I have in a column vector in my FFDF. As an example,…
ModalBro
  • 544
  • 5
  • 25
0
votes
2 answers

How to overload the functions "[<-.ffdf" and "[.ffdf" from package ff (in R)?

I've experienced the write error from ff package. This answer suggests that the solution could be to overload the functions "[<-.ffdf" and "[.ffdf". Could somebody elaborate this in more details, or at least suggest what I need to figure out to…
H13
  • 1
  • 2
0
votes
0 answers

ffsave from ff package creates a large file "clone.ff"

I have a question about ffsave(from ff package). I have a large matrix that I save like this: ffsave(Mat, file=paste(results_wd,"Mat",sep=""), rootpath=results_wd) rm(Mat) gc() In my folder (results_wd) I have "Mat.ffData", "Mat.RData" and…
Chika
  • 1,455
  • 2
  • 16
  • 24
0
votes
1 answer

FF in R: No Applicable Method for 'recodeLevels'

I'm trying to load a huge (~5GB) .csv file into R using read.csv.ffdf. The command goes: npi <- read.csv.ffdf(file="C:/Users/DSA/Dropbox/Team Shared Files/People/Ross/NPI_Parse/Zips/npi_full.csv", VERBOSE=TRUE,…
Ross Wardrup
  • 311
  • 1
  • 9
  • 26
1 2 3
…
10
11