0

I'm using the multiprocessing library in python 3.7.

In my use case, I start with a normal list of dicts - I use this to create a ListProxy object from a SyncManager which I then use in a pool to syncronise the list (created as below).

data = [{...},{...},{...}] # arbitrary list of dicts that I want to sync in a pool

with Manager() as manager:

  list_proxy = manager.list()
  for row in data:
    list_proxy.append(manager.dict(row))

  # ... list_proxy used in pool.starmap later

My question: is there a pythonic way to change my ListProxy back into a normal list once the pool has finished running? I looked through the docs and even into the source code and it seems the only way might be to use the _getvalue() method from the BaseProxy class - but this seems hacky, and I will also have to recursively 'un-proxy' any nested proxy objects.

I need to return the normal list to serialise it to json later in the code, ProxyObjects are not easily serialised.

elembie
  • 600
  • 2
  • 8

1 Answers1

1

I believe you can just call the "real" object function on the proxy object and turn it into the "real" object. See this example of a proxy dict which I share across instances and later turn into a "real" dict in order to serialize it.

def main(state):
    # Do something to input
    dict_proxy['state'] = state

if __name__ == '__main__':
    manager = Manager()
    dict_proxy = manager.dict()

    num_cores = int(2)
    with Pool(num_cores) as p:
       p.map(main, ['AK', 'HI'])

    dict_real = dict(dict_proxy)
cgx
  • 276
  • 2
  • 7