I'm solving a multi objective optimization problem with pymoo and integrated with Dask. In pymoo you have to define a Problem class to pass it to the optimizer, I started with this class
class ChungProblemWeightedDask(ChungProblemDask):
def __init__(self, client: Client, weights: np.ndarray, *args, **kwargs):
super().__init__(client, *args, **kwargs)
self.difference_weights_matrix = np.abs(
np.subtract.outer(weights, weights)
)
self.client = client
def _evaluate(self, X, out, *args, **kwargs):
W = X[:, 0]
X = X[:, 1:]
N = self.occurrence_matrix
P = self.difference_weights_matrix
def F13(ind, H):
return -sum(np.dot(var, np.matmul(H, var)) for var in ind)
X_pop = [self.list_to_matrix(ind, self.n_skus) for ind in X]
jobs = [
*self.client.map(F13, X_pop, H=N),
*self.client.map(F13, X_pop, H=-P),
]
jobs_results = self.client.gather(jobs)
out["F"] = np.column_stack(
[
np.row_stack(jobs_results[0 : len(X_pop)]),
np.row_stack(jobs_results[len(X_pop) :]),
W,
]
)
out["G"] = self.constraints(X_pop, W)
Here the client is a dask.distributed.Client and could be a LocalCluster or distributed across multiples machines (for development I'm using a local one). The class ChungProblemDask is a class where I simply define all the logic of my optimization problem, and I think it's not important for the question I have. The function F13 inside (why inside? look question 2) the _evaluate method is the function to optimmize, and look that I use twice, for the matrix N and P (very large matrices).
First Question
When I solve the optimization problem with the class defined above I have no problems and everything works fine. But look close, the way that I'm solving this, in my opinion, is not optimal because I have to map twice the variable X_pop when I do
jobs = [*self.client.map(F13, X_pop, H=N),*self.client.map(F13, X_pop, H=P)] , being that the function is the same with different matrices only. I tried something like this
def F13(ind):
F1 = 0
F3 = 0
for var in ind:
F1 = F1 + np.dot(var, np.matmul(N, var))
F3 = F3 + np.dot(var, np.matmul(P, var))
return -F1, F3
X_pop = [self.list_to_matrix(ind, self.n_skus) for ind in X]
jobs = self.client.map(F13, X_pop)
jobs_results = self.client.gather(jobs)
When I ran this I threw the error concurrent.futures._base.CancelledError, but impressively if I change the function F13 to this form
def F13(ind):
F1 = 0
F3 = 0
for var in ind:
F1 = F1 + np.dot(var, np.matmul(N, var))
F3 = F3 + np.dot(var, np.matmul(N, var))
return -F1, F3
i.e F1 and F3 are the same (note that I use N twice), I have not problem and everything works fine!. What's going on here? It has to do with the fact of returning a tuple?
Second Question
I can't move the definition of F13 to a staticmethod and get it out from inside the _evaluate method, I got PickleError. I know that the way to distribute the data in the workers goes through a serialization process.
There is a way to do something like this?
class ChungProblemWeightedDask(ChungProblemDask):
def __init__(self, client: Client, weights: np.ndarray, *args, **kwargs):
super().__init__(client, *args, **kwargs)
self.difference_weights_matrix = np.abs(
np.subtract.outer(weights, weights)
)
self.client = client
@staticmethod
def F13(ind, H):
return -sum(np.dot(var, np.matmul(H, var)) for var in ind)
def _evaluate(self, X, out, *args, **kwargs):
W = X[:, 0]
X = X[:, 1:]
N = self.occurrence_matrix
P = self.difference_weights_matrix
X_pop = [self.list_to_matrix(ind, self.n_skus) for ind in X]
jobs = [
*self.client.map(self.F13, X_pop, H=N),
*self.client.map(self.F13, X_pop, H=P),
]
jobs_results = self.client.gather(jobs)
out["F"] = np.column_stack(
[
np.row_stack(jobs_results[0 : len(X_pop)]),
np.row_stack(jobs_results[len(X_pop) :]),
W,
]
)
out["G"] = self.constraints(X_pop, W)
PS This is my first time using dask, any comments that help with optimization are welcome. If something I have written does not make sense, I apologize. If additional information is needed I can provide it.
PS2 The algorithm That I use is the NSGAII. The matrices in the game are huge (4000x4000 app). We use app 15000 iterations and in every iteration we have to calculate app 200 matrix multiplications. Doing this without dask tooks 8 days, and with 4-5 hrs. Running in a single but powerful machine.
