0

I want to run evolutionary algorithms like GA,PSO using pyspark on spark.How to do this using MLLib using Deap python library.Is there any other library available to perform same task.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
yogendrask
  • 21
  • 1
  • 6
  • Possible duplicate of [Using DEAP (genetic algorithm library) with spark](https://stackoverflow.com/questions/45607035/using-deap-genetic-algorithm-library-with-spark) – usernumber Nov 08 '19 at 09:49
  • upvoting for SEO reasons. This question is much easier to find than a very niche and less-known experimental framework called DEAP. Alternatives would also be beneficial since this question is broader than the DEAP question. @usernumber – David Feb 23 '23 at 10:47

2 Answers2

1

Have a look at my answer on how to use DEAP with Spark and see if it works for you.

Here is an example of how to configure the DEAP toolbox to replace the map function by a custom one using Spark.

from pyspark import SparkContext

sc = SparkContext(appName="DEAP")

def sparkMap(algorithm, population):
    return sc.parallelize(population).map(algorithm)

toolbox.register("map", sparkMap)
frenchoverflow
  • 131
  • 5
  • 1
    What is `toolbox`? Please don't duplicate answers. Provide one good one, then link to it via the comments – OneCricketeer Sep 25 '17 at 01:35
  • It is an utility class provided by DEAP to make it easy to change operators without impacting the rest of your algorithm (docs [here](http://deap.readthedocs.io/en/master/tutorials/basic/part2.html#using-the-toolbox). In my example, it replaces the map function (used to map the fitness function to all the individiduals) by a custom one using Spark parallelize function. You can find another example in the [docs](http://deap.readthedocs.io/en/master/tutorials/basic/part4.html). – frenchoverflow Sep 25 '17 at 01:47
0

In https://github.com/DEAP/deap/issues/268 they show how to do this in the DEAP package. However this is an issue. but they mention there is a pull request (https://github.com/DEAP/deap/pull/76), and is seems the fixed code/branch is from a forked repo.
It sounds like if you rebuild the package with that code it should resolve the issue.

Another resource I found, haven't tried it, is https://apacheignite.readme.io/docs/genetic-algorithms. Also came across this https://github.com/paduraru2009/genetic-algorithm-with-Spark-for-test-generation

Bernie Lindner
  • 61
  • 1
  • 3