I have a function that gets some kwargs, does some calculations and returns a dict of vectors. I execute this function a few thousand times in a process pool and aggregate the results afterwards. The problem is that the intermediate results stay in memory until all workers are done, which fills up memory and leads to an oom error.
Approach: As the tasks are embarrassing parallel (independent of each other), I thought it might be possible to aggregate worker specific results, return one aggregated result per worker and further aggregate those in the main process. With this I would only have one result per worker occupying memory, while the sub results can be discarded earlier.
Question: I thought of using the initializer function of concurrent.futures.ProcessPoolExecutor to initialize a worker specific variable for intermediate results, but I haven't figured out how to return this variable when all jobs are done (the pool gets closed). Is there a "finalizer function" for workers, which can be used for returning the initialized "worker result"? Or is there a better way to achieve this?
I looked at this question and was wondering, whether this is the only way to solve the problem, especially if the parameters are not known before/are generated on the fly.
