R doRedis package: parallel computation in R using Redis server database and the foreach %dopar% construct.
R doRedis package: parallel computation in R using Redis server database and the foreach %dopar% construct.
Questions tagged [r-doredis]
15 questions
15
votes
1 answer
How does doRedis work?
I've been playing around with the R interface to the redis database, as well as the doRedis parallel backend for foreach. I have a couple of questions, to help me better apply this tool:
doMC, doSMP, doSnow, etc. all seem to work by calling up…
Zach
- 29,791
- 35
- 142
- 201
5
votes
1 answer
How to install doRedis package version 1.0.5 into R 3.0.1 on Windows?
There is a bug in the current latest version of doRedis (1.1.0), and the downgrade to version 1.0.5 is recommended. However, I have not been able to install the 1.0.5 package into R 3.0.1 on Windows XP!
I tried to follow the answers of the question…
Tomas
- 57,621
- 49
- 238
- 373
5
votes
1 answer
Error occurring in caret when running on a cluster
I am running the train function in caret on a cluster via doRedis. For the most part, it works, but every so often I get errors at the very end of this nature:
error calling combine function:
Tommy Levi
- 771
- 5
- 12
3
votes
1 answer
How to use doRedis and caret on a cluster?
I've been playing around with the doRedis package in R to try run some code on a cluster. I've got one Windows machine and one machine running Ubuntu (which is where redis is installed).
I can happily run the example from the doRedis documentation…
Flobagob
- 76
- 1
- 12
3
votes
1 answer
doRedis on Windows 7 gives error as soon as foreach loop is run
I have a simple setup as follows:
Sys.info()
sysname release version nodename
"Windows" "7 x64" "build 7601, Service Pack 1" …
intelinsight
- 63
- 5
2
votes
0 answers
How to manually add a task to a redis queue for a redisWorker?
Is it possible to manually add a task to the redis queue, so that it can be executed by a redis worker?
As a simple example, I'm launching a worker using:
require('doRedis')
redisWorker('jobs')
In another R session, I'm creating a queue and would…
nassimhddd
- 8,340
- 1
- 29
- 44
2
votes
1 answer
doRedis workers shut down immediately on windows
Solved -- look below.
I'm trying to run a doRedis parallel backend with R on Windows 7. And I'm running into problems while trying the first example from the doRedis vignette:
library('doRedis')
registerDoRedis('jobs')
startLocalWorkers(n = 2, queue…
orr
- 35
- 5
1
vote
0 answers
Can foreach selectively utilize multiple back-ends (doParallel and doRedis) at the same time?
I am processing files which are about 1 Gb in size. The processing can be broken into two steps, one which depends on the large files and a second which depends on a smaller amount of data. Can I queue some foreach commands for the doParallel back…
nate
- 927
- 1
- 7
- 19
1
vote
2 answers
TryCatch and names Error when running doRedis examples from documentation
I am trying to use doRedis to create an R cluster and parallel process some of my calculations with foreach, but keep running into errors. My code works with foreach %do% but when I try to run on the cluster with %dopar% it fails. I also tried…
Nathan Wadhwani
- 23
- 5
1
vote
1 answer
Running doRedis- Object not found even when it's been exported
I'm testing the doRedis package by running a worker one machine and the master/server on another. The code on my master looks like this:
#Register ...
r <- foreach(a=1:numreps, .export(...)) %dopar% {
train <- func1(..)
best <-…
rw.liang1
- 418
- 4
- 12
1
vote
1 answer
doRedis return errors in windows 8 x64 for different Redis server build
I am using the Redis server from the link :
http://cloud.github.com/downloads/rgl/redis/redis-2.4.6-setup-64-bit.exe
with R version3.0.3, doRedis 1.1.0, rredis 1.6.8
The Redis worker end immediately after receiving jobs
>…
Ravat
- 207
- 2
- 8
0
votes
1 answer
R - doRedis - Overwrite getTask to control the order of execution in parallel foreach loops
Problem: I need to control the order of execution in which tasks are processed in parallel by a foreach loop. Unfortunately, this is not supported by foreach.
Solution in mind: Using doRedis to use the database to hold all tasks, that are executed…
Exocom
- 791
- 2
- 8
- 24
0
votes
2 answers
Why don't my doRedis workers begin processing until all of the jobs are in the redis server
When using foreach and doRedis the doRedis workers wait until all jobs have reached the redis server before beginning processing. Is it possible to have them begin before all the preprocessing has finished?
I am using an iterator which is working…
nate
- 927
- 1
- 7
- 19
0
votes
1 answer
How to recognize and kill all threads started in R session
Sometimes threads are started by R package (e.g. in doRedis, see related code). Is there a way how can I list threads spawned within current R session and kill these threads within R? Or can I recognize and kill them in Windows task list?
I am using…
Tomas
- 57,621
- 49
- 238
- 373
0
votes
1 answer
doRedis with strange socket connection error in Ubuntu Linux, R, and RStudio
I am trying to get doRedis R package working in my latest Ubuntu 10.0.4 environment with R 2.12.1. I tried download the latest 2.15.1 but it installs (with apt-get install...) this other version. Anyhow, when I run the following lines from doRedis.R…
heavy rocker dude
- 2,271
- 8
- 33
- 47