5

I have a generator function (Python) that works like this

def Mygenerator(x, y, z, ...):
    while True:
        # code that makes two matrices based on sequences of input arrays
        yield (matrix1, matrix2)

What I want to do is to add the output from this generator. This line does the job:

M1, M2 = reduce(lambda x, y: x[0] + y[0], x[1] + y[1], Mygenerator(x, y, z, ...))

I would like to parallelize this to speed up the computations. It is important that the outputs from Mygenerator is reduced as it is yielded, since list(Mygenerator(...)) would take too much memory.

smac89
  • 39,374
  • 15
  • 132
  • 179
sulkeh
  • 935
  • 7
  • 21
  • `x[0] and y[0]` are numbers or something else? – smac89 Nov 10 '14 at 07:15
  • x[0] and y[0] are matrices. The generator returns (matrix1, matrix2), and the reduce operation adds together the matrix1's and matrix2's from consecutive calls. – sulkeh Nov 10 '14 at 08:18
  • Not sure what you are trying to do here. Did you mean to have parenthesis around the lambda like: `reduce((lambda x, y: x[0] + y[0], x[1] + y[1]), Mygenerator(x, y, z, ...))`. I wont even comment on `while True` in the generator ;) –  Nov 11 '14 at 16:12
  • I'm trying to iterate over segments of x, y, and z (etc.) and compute matrix1 and matrix2 for each segment. The lambda expression returns the sum of the `matrix1`s and `matrix2`s. I have found a solution, which is posted below – sulkeh Nov 11 '14 at 20:25

1 Answers1

3

To answer my own question, I found a solution that seems to work as I had hoped:

First, Mygenerator is no longer a generator but a function. Also, instead of looping through segments of x, y and z, I now pass one segment to the function at the time:

def Myfunction(x_segment, y_segment, z_segment):
        # code that makes two matrices based on input arrays
        return (matrix1, matrix2)

Using multiprocessing.Pool with the imap (generator) function seems to work:

pool = multiprocessing.Pool(ncpus)
results = pool.imap(Myfunction, 
                    ( (x[i], y[i], z[i]) for i in range(len(x)) )
M1, M2 = reduce(lambda r1, r2: (r1[0] + r2[0], r1[1] + r2[1]), 
                    (result for result in results))
pool.close()
pool.join()

where I changed the x and y in the lambda expression to r1 and r2 to avoid confusion with the other variables with the same name. When trying to use a generator with multiprocessing I got some trouble with pickle.

The only disappointment with this solution is that it didn't really speed up the computations that much. I guess that has to do with overhead operations. When using 8 cores, the processing speed was increased by approximately 10%. When reducing to 4 cores the speed was doubled. This seems to be the best I can do with my particular task, unless there is some other way of doing the parallelizing...

The imap function was necessary to use here, since map would store all the returned values in memory before the reduce operation, and in this case that would not be possible.

sulkeh
  • 935
  • 7
  • 21