Using multiprocessing to call instances of a class method from another class

Question

I need to use a multiprocessing technique in a class that call a method of another class. Below the explanation.

I am coding a multi-agent strategy. So, my code is structured in 3 classes:

1. agent class
2. environment class
3. main.

In the agent class, I am running a MILP problem that return a dictionary of decision variables:

class Agent:
    def __init__():
        ....
    def Optimisation():
       .....
       return decision_agent

The class environment receives the output from all the agents to compute the 'community' output. Since every agent is an independent entity, agents will be computing their optimisation problem in parallel. Therefore, I want to use multiprocessing in the environment class to get the decisions of all agents

import Agent as Agent

class Env:
    def __init__():
         self.agents = #list of agents in my environment

    def community_output():
        #**start the part of code I want to multiprocess**
        for agent in self.agents:
            agents_output.append(agent.Optimisation) 
        #**end the part of code I want to multiprocess**

        """compute community_output based on agents_output """

        return community_output

Then in the main, call the method community_output():

import Env as Env

if __name__ == '__main__': 
    env = Env()
    result = env.community_output()

How can I proceed please?

Since I have a list of instance method as an iterable, I tried the following:

def job(instance):
    instance.Optimisation()
    return instance

if __name__ == '__main__':
   pool = mp.Pool(processes=4)
   agents_output= pool.map(job, agents)
   pool.close() 
   pool.join()

This only works if I remove the Env class and put all its processing before def job(instance).

When I integrate it in the Env class:

import Agent as Agent

class Env:
    def __init__():
         self.agents = #list of agents in my environment

    def job(instance):
        instance.Optimisation()
        return instance

    def community_output():
        pool = mp.Pool(processes=4)
        agents_output = pool.map(self.job, self.agents)
        pool.close() 
        pool.join()

        """ Compute community_output based on agents_output """

        return community_output

Import Env as Env
if __name__ == '__main__': 
    env = Env()
    result = env.community_output()

I got the error:

  File "C:\Users\chaim\...\Env.py", line 576, in community_output
    results = pool.map(self.job, self.agents)
  File "C:\Users\chaim\anaconda3\envs\picos\lib\multiprocessing\pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\chaim\anaconda3\envs\picos\lib\multiprocessing\pool.py", line 657, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '[{'opt_prob': <Mixed-Integer Linear Program>, 'q_us_val': <190x1 matrix, tc='d'>, 'q_ds_val': 0.0, 'p_net_exp_val': <48x1 matrix, tc='d'>, 'q_sup_sell_val': <48x1 matrix, tc='d'>, 'q_sup_buy_val': <48x1 matrix, tc='d'>, 'p_es_ch_val': <48x1 matrix, tc='d'>, 'p_es_dis_val': <48x1 matrix, tc='d'>, 'q_ds_0': <1×1 Real Constant: 0>, 'q_us_0': <1×1 Real Constant: q_usᵀ·[190×1]>}]'. Reason: 'PicklingError("Can't pickle <function <lambda> at 0x000001B8B9C31288>: attribute lookup <lambda> on picos.modeling.options failed")'

I note that:

{'opt_prob': <Mixed-Integer Linear Program>, '... q_usᵀ·[190×1]>}

is the dictionary returned by the method agent.Optimisation and picos is the optimisation API used.

Any clue how can I solve this?

Answer 1

are you trying to call Optimisation() from the Agent Module? If So then shouldn't

def job(instance):
    instance.Optimisation()
    return instance

instead be

def job():
    return Agent.Optimisation()

Answer 2

I'm not familiar with the PICOS per se, but after making a few assumptions about it I think the following is a general outline of how you could do what you want. Note that the function arument being passed to pool.map() is the unbound Agent class method optimisation(). It will be called with one of the Agent instances you have in the list of them.

import multiprocessing as mp
from random import randrange  # For testing purposes.


class Agent:
    def __init__(self):
        ...
    def optimisation(self):
       decision_agent = {'var1': randrange(10), 'var2': randrange(10), 'var3': randrange(10)}
       return decision_agent


class Env:
    def __init__(self, num_agents):
         self.agents = [Agent() for _ in range(num_agents)]

    def community_output(self):
        with mp.Pool(processes=4, maxtasksperchild=1) as pool:
            agents_output = pool.map(Agent.optimisation, self.agents)
        return agents_output


if __name__ == '__main__':
    env = Env(3)
    result = env.community_output()
    print(result)