4

Question

I'm looking for an embedded IPC framework because I want to achieve the following parallelism:

enter image description here

  • I want to analyse numerous sources of live data
  • I plan to use a new process for each data source to poll for updates
  • This allows each data source to be updated asynchronously at its own polling interval, and without blocking updates for other data sources (important if a particular server is very slow)
  • I need an IPC framework so I can fetch the updates from each process that is doing the polling to the process in which I'm doing analysis
  • All R processes run on Windows ≥ XP without admin privileges, hence MPI as an IPC framework is not a possibility

To start, we can consider only the case where all processes run on a single machine (i.e. not over network). Does such a framework exist?

Additional info

I've taken a look at R's HPC page, which mentions MPI and NetworkSpaces as possible frameworks that can handle inter-process communication. However, both require installations of external .exes.

The HPC page also mentions snow, but that package only seems to allow parallelisation via functions like sfLapply which does not allow for fine-enough control of the message passing.

mchen
  • 9,808
  • 17
  • 72
  • 125
  • This is a bit unclear still. Do you see one master computer spawning these threads to possibly other PCs? What does 'distribute solutions' mean? How are the answers combined - you say "fetch the updates" but maybe the threads can push them to a waiting service? Just a bit unclear. Maybe draw a diagram. – Spacedman Feb 07 '14 at 21:23
  • OK, will do - diagram coming soon. Let me get some sleep first. – mchen Feb 08 '14 at 02:02

0 Answers0