Why does my parallel traversal Haskell program leak memory?

Question

Consider the following Haskell program (I'm doing this mostly for learning purposes):

import qualified Control.Concurrent.MSem as Sem
import System.Environment (getArgs)
import Control.Concurrent (forkIO)
import Control.Monad

-- Traverse with maximum n threads
parallelTraverse :: Foldable a => Int -> (b -> IO()) -> a b -> IO ()
parallelTraverse n action values = do
  sem <- Sem.new n
  forM_ values $ \value -> Sem.with sem (forkIO $ action value)

main :: IO ()
main = do
  args <- getArgs
  let nThreads = read . head $ args :: Int
  parallelTraverse nThreads print [(1::Int)..]

when I run it, memory quickly climbs to several GB. I tried various combinations to make sure I discard the results of intermediate computations (the print actions). Why is it still leaking space?

You have not prevented an unbounded number of threads from running simultaneously. Read your program carefully. — Reid Barton, Sep 07 '15 at 20:08
@ReidBarton : I think I get it: forkIO returns immediately, making the semaphore useless. Back to the design table :) — static_rtti, Sep 07 '15 at 20:12

Nikita Volkov · Accepted Answer · 2015-09-07T20:23:40.543

First of all, you have an evident mistake in the following piece:

Sem.with sem (forkIO $ action value)

You're addressing the semaphore from the master thread around the "fork" operation instead of the action there. Following is the proper way to implement it:

forkIO (Sem.with sem (action value))

I.e., to address the semaphore from the context of the forked thread.

Secondly, in the following code you're calling the parallelTraverse operation on an infinite list:

parallelTraverse nThreads print [(1::Int)..]

Which results in the infinite forking of threads. And since the forkIO operation is roughly instantaneous for the calling thread, it's pretty much no surprise that you're running out of resources quite soon.

To use the semaphore to limit the number of worker threads the with pattern simply won't do in your case. Instead you should use the explicit combination of wait and signal and not forget to treat the exceptions properly (in case you expect them). E.g.,:

parallelTraverse :: Foldable a => Int -> (b -> IO()) -> a b -> IO ()
parallelTraverse n action values = do
  sem <- Sem.new n
  forM_ values $ \value -> do
    Sem.wait sem
    forkIO $ finally (action value) (Sem.signal sem)

My goal is to limit the number of concurrent threads, but now I see why it doesn't work. The infinite list is by design, because I'd like my program to be able to handle an infinite stream of actions. — static_rtti, Sep 07 '15 at 20:13
Thanks a lot. I just needed a little extra help to understand that I needed wait and signal on the two sides of forkIO, which is why (if I understand correctly) with can't be used in this case. — static_rtti, Sep 07 '15 at 20:41

Why does my parallel traversal Haskell program leak memory?

1 Answers1