4

In haskell, given I have a function call in the following form: foo a b, where a doesn't depend on b and vice versa. It seems that it could be automatically detected that a and b can be evaluated in parallel, but this doesn't seem to be the case in GHC. Instead constructs like par need to be used to denote what can be evaluated in parallel.

So, why can't parallelization in haskell just happen automatically? Or if it already does, why do constructs like par exist?

Don Stewart
  • 137,316
  • 36
  • 365
  • 468
Kritzefitz
  • 2,644
  • 1
  • 20
  • 35
  • 3
    The problem with automatic parallelism is that it adds overhead and it's very difficult to automatically determine if parallel evaluation of two expressions brings benefit or not. – Petr Sep 28 '14 at 11:59
  • See also [Data Parallel Haskell](http://www.haskell.org/haskellwiki/GHC/Data_Parallel_Haskell). – Petr Sep 28 '14 at 12:00
  • @PetrPudlák So the problem isn't as much that the principle doesn't work, but that it's not such a big gain or might even become slower? – Kritzefitz Sep 28 '14 at 12:12

1 Answers1

5

It seems that it could be automatically detected that a and b can be evaluated in parallel

Parallelism can be detected automatically, as you hint at, by looking at the dependencies between values. This is particuarly easy when there are no side-effects involved.

The problem is that is knowing when to stop making things parallel.

It all boils down to knowing at compile time how much work will occur at runtime. These "cost models" are hard to do in general for arbitrary code.

Consider:

  • should every argument to (+) be evaluated in parallel?
  • should every map be evaluated in parallel?

If we naively parallelize all independent computations, the compiler will generate a huge amount of parallel tasks. Millions or billions of parallel expressions. Which our 8 or 16 core machines are just not ready to handle.

Naive parallelization results in massive overheads trying to schedule work onto the small amount of available parallel hardware.

This gap between the amount of parallelism in pure programs, and the available hardware, forces us to make some compromises. Namely:

  1. user-annotated hints of which things are costly enough to do in parallel
  2. subsets of the language that have a clear cost model, so the compiler can be smart.

Examples of the first form -- user hints -- are par annotations or the Par monad. Of the second -- automatically parallel sub-languages -- see Data Parallel Haskell.

Don Stewart
  • 137,316
  • 36
  • 365
  • 468
  • 3
    This answer *presumes* that either (a) there is a very naive runtime environment, spawning tasks blindly and/or (b) it must be known at compile time how much parallelism must be introduced where. It does not point out that automatic parallelism (in Haskell) is just an unsolved problem, not an *unsolvable* one. (a) Why can't e.g. the (fixed pool of) worker threads determine themselves where they could go and do some work, based on the runtime stack of existing threads? (b) Why does it have to be known at compile time where parallelization must occur? – masterxilo Apr 13 '17 at 00:10
  • 2
    I don't see why we could not e.g. write a Haskell-style functional (side effect & state free) program in C/C++ and just use e.g. [Cilk](https://en.wikipedia.org/wiki/Cilk)'s `spawn` and `sync` keywords blindly on every sub-function call (`a` and `b` in the `foo a b` example) and then `sync` before computing `foo a' b'` on the results. The keywords are mere hints and the runtime ignores them if all worker threads are already busy. Seems like the ideal solution... – masterxilo Apr 13 '17 at 00:14