21

I've read a lot of articles about distributed Haskell. Much work has been done but seems to be in the area of distributing computations. I saw the remote package which seems to implement Erlang-style messaging passing but it is 0.1 and early stage.

I'd like to implement a system where there are many separate processes that provide distinct services, and are tied together by several main processes. This seems to be a natural fit for Erlang, but not so for Haskell. But I like Haskell's type safety.

Has there been any recent adoption of Erlang-style process management in Haskell?

  • 3
    Like you say, it seems to be a natural fit for Erlang--isn't that the type of problem Erlang was *designed* for? I like Haskell plenty but this sounds like a pretty clear "right tool for the job" situation. Why not just use Erlang? – C. A. McCann Dec 02 '11 at 21:54
  • 4
    Because there are other considerations than concurrency, and I believe Haskell provides better benefits in these areas. What I am looking for is a recommendation on how to best do process-oriented concurrency in Haskell. –  Dec 02 '11 at 22:29

2 Answers2

16

If you want to learn more about the remote package, a.k.a CloudHaskell, see the paper as well as Jeff Epstein's thesis. It aims to provide precisely the actor abstraction you want, but as you say it is in the early stages. There is active discussion regarding improvements on the parallel-haskell mailing list, so if you have specific needs that remote doesn't provide, we'd be happy for you to jump in and help us decide its future directions.

More mature but lower-level than remote is the haskell-mpi package. If you stick to the Simple interface, messages can be sent containing arbitrary Serialize instances, but the abstraction is still way lower than remote.

There are some experimental systems, such as described in Implementing a High-level Distributed-Memory Parallel Haskell in Haskell (Patrick Maier and Phil Trinder, IFL 2011, can't find a pdf online). It blends a monad-par approach of deterministic dataflow parallelism with a limited ability to make the I-structures serializable over the network. These sorts of abstraction have promise for doing distributed computation, but since the focus is on computing purely-functional values rather than providing Erlang-style processes, they probably wouldn't be a good fit for your application.

Also, for completeness, I should point out the Haskell wiki page on cloud and HPC Haskell, which covers what I describe here, as well as the subsection on distributed Haskell, which seems in need of a refresh.

acfoltzer
  • 5,588
  • 31
  • 48
  • 3
    The thesis and paper for CloudHaskell use a lot of terminology and "keywords" (function names, data types) which are different from the CloudHaskell/remote package. Someone writing a more up-to-date explanation/tutorial would be *very* helpful! – amindfv Dec 24 '11 at 06:20
  • 1
    Apparently CloudHaskell is being re-implemented: https://github.com/haskell-distributed/distributed-process – balu Jul 27 '12 at 11:12
8

I frequently get the feeling that IPC and actors are an oversold feature. There are plenty of attractive messaging systems out there that have Haskell bindings e.g. MessagePack, 0MQ or Thrift. IMHO the only thing you have to add is proper addressing of processes and decide who/what is managing this addressing capability.

By the way: a number of coders adopt e.g. 0MQ into their Erlang environments, simply because it offers the possibility to structure messaging via message brokers rather then relying on pure process to process messaging in super scale.

In a "massively multicore world" I personally assume that shared memory approaches will eventually be outperforming messaging. Someone can then always come and argue with asynchrony of course. But already when you write that you want to "tie together" your processes by "several main processes" you in fact speak about synchronization. Also, you can of course challenge whether a single function, process or thread is the right level of parallelization.

In short: I would probably see whether MessagePack or 0MQ could fit my needs in Haskell and care for the rest in my code.

J Fritsch
  • 3,338
  • 1
  • 18
  • 40
  • 1
    Good points, but I'd add that shared memory and messaging aren't competing, but complementary models. MPI and OpenMP are frequently combined in ways that play to each models' strengths. I'm working right now on something similar with the `Par` monad, where shared memory parallelism runs computations which might be sent between nodes with message passing. – acfoltzer Dec 05 '11 at 14:31
  • 1
    Certainly right. However, I think that e.g. 10G Ethernet is meanwhile good enough to make shared memory approaches viable beyond the boundaries of the physical system. Look at [RoCe] (http://dl.acm.org/citation.cfm?id=1634475) and [iWARP] (http://en.wikipedia.org/wiki/IWARP). I hope that I can win a commercial sponsor for a testbed. – J Fritsch Dec 05 '11 at 20:25
  • For those of us who are new to all this, what do you mean by "shared memory approaches"? – jberryman Dec 09 '11 at 21:05
  • @jberryman A very good rundown skimming a number of issues very critically is here: http://bartoszmilewski.wordpress.com/2010/08/02/beyond-locks-and-messages-the-future-of-concurrent-programming/ – J Fritsch Dec 09 '11 at 21:28