3

Is IT possible to use DEAP ( http://deap.readthedocs.io/en/master/) with a spark cluster to map the fitness evaluation function. I would like to run a GA but the fitness function is rather long and I was planning on distributing it on a spark cluster.

secretgenes
  • 1,291
  • 1
  • 19
  • 39
Julien Denos
  • 79
  • 1
  • 9

1 Answers1

2

You should look at the Using Multiple Processors section in the DEAP documentation and at this example. They explain how to replace the map function in the DEAP toolbox by a map function of your choice.

To use pyspark to map the fitness evaluation function, you could do something like that:

from pyspark import SparkContext

sc = SparkContext(appName="DEAP")

def sparkMap(algorithm, population):
    return sc.parallelize(population).map(algorithm)

toolbox.register("map", sparkMap)
frenchoverflow
  • 131
  • 5