Distributing Scala over a cluster?

Question

So I've recently started learning Scala and have been using graphs as sort of my project-to-improve-my-Scala, and it's going well - I've since managed to easily parallelize some graph algorithms (that benefit from data parallelization) courtesy of Scala 2.9's amazing support for parallel collections.

However, I want to take this one step further and have it parallelized not just on a single machine but across several. Does Scala offer any clean way to do this like it does with parallel collections, or will I have to wait until I get to the chapter in my book on Actors/learn more about Akka?

Thanks! -kstruct

om-nom-nom · Accepted Answer · 2012-11-07T08:49:00.687

11

There was an attempt of creating distributed collections (currently project is frozen).

Alternatives would be Akka (which recently got really cool addition: Akka Cluster), that you've already mentioned, or full-fledged cluster engines, that is not parallel collections in any sense and more like distributing cluster over the scala but could be used in your task in some way - such as Scoobi for Hadoop, Storm or even Spark (specifically, Bagel for graph processing). There is also Swarm that was build on top of delimited continuations. Last but not least is Menthor - authors claiming that it is especially fits graph processing and makes use of Actors.

Since you're aiming to work with graphs you may also consider to look at Cassovary that was recently opensourced by twitter.

Signal-collect is a framework for parallel dataprocessing backed with Akka.

edited Nov 07 '12 at 08:49

answered Mar 11 '12 at 05:27

om-nom-nom

62,329
13
183
228

Check out [Scrunch](https://github.com/cloudera/crunch/tree/master/scrunch) as well for Hadoop – dhg Mar 11 '12 at 07:52
Thanks for the note on signal-collect! – adelbertc Apr 08 '12 at 05:38
1

Scalding looks awesome actually, will probably end up using that. – adelbertc Apr 15 '12 at 06:26

score 2 · Answer 2 · answered Mar 11 '12 at 05:28

You can use Akka ( http://akka.io ) - it has always been the most advanced and powerful actor and concurrency framework for Scala, and the fresh-baked version 2.0 allows for nice transparent actor remoting, hierarchies and supervision. The canonical way to do parallel computations is to create as many actors as there are parallel parts in your algorithm, optionally spreading them over several machines, send them data to process and then gather the results (see here).

Distributing Scala over a cluster?

2 Answers2