Stack Overflow
Stack Overflow

Questions tagged [snowfall]

Usability wrapper around snow for easier development of parallel R programs.

66 questions
4
votes
1 answer

R: making cluster in doParallel / snowfall hangs

I've got two servers on a LAN with fresh installs of Centos 6.4 minimal and R 3.0.1. Both computers have doParallel, snow, and snowfall packages installed. The servers can ssh to each other fine. When I attempt to make clusters in either direction,…
dlv
  • 557
  • 1
  • 6
  • 14
3
votes
1 answer

Create line density per polygon using tidyverse + sf R

I have a GIS question that has been stumping me for some time now. The end goal would be to extract the density of lines per pixel/voxel/polygon using tidyverse/sf packages. As of now I have a function that works when I execute line-by-line, but not…
nate-m
  • 557
  • 3
  • 14
3
votes
0 answers

R Data processing performance (snowfall package and function scope)

I'm somewhat new in the R programming world and I'm dealing with some issues related to the parallelization of the processing of (not so much) big data. To this end, I'm using the data.table package for data storage and handling, and the snowfall…
zek
  • 73
  • 5
3
votes
1 answer

Running functions in parallel in R

I have two functions fun1 and fun2. I would like to run them in parallel in my R script. When they complete I'd like to use their output in further processing sequentially. I guess this means that I need to wait for them to complete. I would…
user2630162
  • 137
  • 1
  • 12
3
votes
1 answer

When do I need to use sfExport (R Snowfall package)

I am using snowfall for parallel computing. I am always on only one machine with multiple CPUs (>20 cores). I am processing a large amount of data (>20gb). sfExport() takes very long. When I run my test codes on my laptop and check the CPU usage, it…
kn1g
  • 358
  • 3
  • 16
3
votes
1 answer

R parallel computing with snowfall - writing to files from separate workers

I am using the snowfall 1.84 package for parallel computing and would like each worker to write data to its own separate file during the computation. Is this possible ? if so how ? I am using the "SOCK" type connection e.g., sfInit( parallel=TRUE,…
user1762374
  • 31
  • 1
2
votes
2 answers

How to initialize libraries by their string names in cluster?

I want to initialize libraries in cluster by their names represented as strings. This code works fine: library(snowfall, rlecuyer, rsprng) sfInit(parallel = TRUE, cpus = 4, type = "SOCK") sfClusterEval(library(e1071)) And this code produces en…
DrDom
  • 4,033
  • 1
  • 21
  • 23
2
votes
0 answers

Can not makeCluster on windows 10

I can not use parallel::makeCluster or snow::makeCluster on a windows 10 computer. I frequently receive this error, each time with a different port number: Error in socketConnection("localhost", port = port, server = TRUE, blocking = TRUE, : …
Ahmed El-Gabbas
  • 398
  • 3
  • 10
2
votes
0 answers

Parallel estimation of multiple nonparametric models using np and snowfall

I am trying to estimate multiple nonparametric models using snowfall. So far I had no problems, but now I run into a problem that I feel unable to resolve. In the MWE below we simply estimate only one model on one node. In my application the…
Martin Schmelzer
  • 23,283
  • 6
  • 73
  • 98
2
votes
1 answer

R snowfall parallel, Rscript.exe goes inactive one by one with time

I am using sfApply in R snowfall package for parallel computing. There are 32000 tests to run. The code is working fine when starting the computing, it will create 46 Rscript.exe processes and each Rscript.exe has a 2% cpu usage. The overall cpu…
yan
  • 21
  • 1
2
votes
0 answers

Scope of variables in parallel R package snowfall

As my understanding, the slave processes cannot access variables in the global/parent environments in R package snowfall. For example, the code below will give an error message as I expected: parfun <- function(i, var1, var2) { …
Bangyou
  • 9,462
  • 16
  • 62
  • 94
2
votes
2 answers

Running different programs on multiple cores

Hopefully this is a simple question I've overlooked an answer to, but in my searches this morning I've been fruitless. I have four cores on my computer, and want to concurrently run different programs (loops) in R over multiple cores. If I open up…
user1399311
  • 271
  • 2
  • 12
1
vote
3 answers

create particles without opengl and cocos2D. snowfall application

well I wanted to create particles like (snow ) without openGL or cocs2D, and I found this sample code called snowfall and in this code there is this : flakeImage = [UIImage imageNamed:@"flake.png"]; // start a timet that will fire 20 times per…
jean bernard
  • 257
  • 2
  • 3
  • 10
1
vote
0 answers

TryCatch and sfApply not producing the same output as TryCatch and Apply

I'm using a 4 parameter log logistic curve to model a time series. Since some time trends were not converging fast enough, and so were producing errors, I used TryCatch so that the loop wouldn't break and I could still get the results for the…
lomoshome
  • 11
  • 1
1
vote
0 answers

Use save() within sfClusterCall()

I want to run a parallelized simulation using snowfall. Trying to aquire data using return() causes the cluster to exceed memory limitations. Data is at some point not recorded anymore. So I want to use save() to write the data to a file after each…
Hope
  • 143
  • 11
1
2
3 4 5