0

I have number of "sites" (m) that each has to process an event (chunks of data. all available on the get go). Each event (n of them) is sent to each site for processing. So you may think that I have nxm tasks. The order of processing is not important, only that one site may not process more than one event at a time (so Task(m,x) cannot run in parallel to Task(m,y))

Currently it's implemented using "OMP parallel for" on the sites, nested in a regular for loop on the events

for(...event...)
#pragma omp parallel for
    for(...site...)
        site.process(event)

This is working fine, however not all sites have the same complexity for each event. i.e all sites have to wait for the slowest site before moving on to the next event. I guesstimate that if I allow workers to move on to the next event I can save a factor of two.

What is the best way to implement this? I'm using C++ I'm looking into TBB Flow Graph, or multiple pipe lines...

One more consideration is that each "event" has to be read from disk, and takes up a bit of memory. Although not critical yet, I would like to have as few event in the system at a time (or limit them). In the current implementation I have only one (plus a couple being prepared in the background) Thanks

Anton
  • 6,349
  • 1
  • 25
  • 53
aivision2020
  • 619
  • 6
  • 14
  • You haven't mentioned any dependencies. can Site A finish all tasks 1..n before Site B start? – NiRR Sep 09 '14 at 13:06
  • yes. Maybe dependencies is not the best word. Limit to parallelization...? So, yes, I could have each site process all the event and then move on the next site. This is just reversing the order of the 2 for loops (leaving the inner one with the parallel for) however I'm not crazy about this solution because of the overhead of preparing (reading) all the events m times. – aivision2020 Sep 09 '14 at 13:10
  • do you still need the answer about this? – Anton Jan 30 '15 at 14:09
  • Yes. I was hoping this is a standard problem handled in tbb and would just require defining the constraints. However if I have to write the manager and allocate and release resources it was postponed to later. – aivision2020 Jan 31 '15 at 14:48

1 Answers1

0

I would use a manager process that keeps track of the processed events of each side (m*n bool matrix ) and events that are currently "in use".

Every site starts with a "random" event. The manager cycles through the sites to check if they finished the current event and assigns a new one if possible.

C++11 provides std::async for such tasks.

Each process call is done async and you can cycle through the corresponding future elements to check if they finished ( wait_for ).

H4kor
  • 1,552
  • 10
  • 28