5

How can I read multiple files as a single ByteString lazily with constant memory? 

readFiles :: [FilePath] -> IO ByteString

I currently have the following implementation but from what I have seen from profiling as well as my understanding I will end with n-1 of the files in memory. 

readFiles = foldl1 joinIOStrings . map ByteString.readFile
    where joinIOStrings ml mr = do
                                l <- ml
                                r <- mr
                                return $ l `ByteString.append` r

I understand that the flaw here is that I am applying the IO actions then rewrapping them so what I think I need is a way to replace the foldl1 joinIOStrings without applying them.

Ben Doerr
  • 1,655
  • 1
  • 13
  • 23

2 Answers2

8

How can I read multiple files as a single ByteString lazily with constant memory?

If you want constant memory usage, you need Data.ByteString.Lazy. A strict ByteString cannot be read lazily, and would require O(sum of filesizes) memory.

For a not too large number of files, simply reading them all (D.B.L.readFile reads lazily) and concatenating the results is good,

import qualified Data.ByteString.Lazy as L

readFiles :: [FilePath] -> IO L.ByteString
readFiles = fmap L.concat . mapM L.readFile

The mapM L.readFile will open the files, but only read the contents of each file when it is demanded.

If the number of files is large, so that the limit of open file handles allowed by the OS for a single process could be exhausted, you need something more complicated. You can cook up your own lazy version of mapM,

import System.IO.Unsafe (unsafeInterleaveIO)

mapM_lazy :: [IO a] -> IO [a]
mapM_lazy [] = return []
mapM_lazy (x:xs) = do
              r <- x
              rs <- unsafeInterleaveIO (mapM_lazy xs)
              return (r:rs)

so that each file will only be opened when its contents are needed, when previously read files can already be closed. There's a slight possibility that that still runs into resource limits, since the time of closing the handles is not guaranteed.

Or you can use your favourite iteratee, enumerator, conduit or whatever package that solves the problem in a systematic way. Each of them has its own advantages and disadvantages with respect to the others and, if coded correctly, eliminates the possibility of accidentally hitting the resource limit.

Daniel Fischer
  • 181,706
  • 17
  • 308
  • 431
  • Thanks, I should have mentioned that I was already using Data.ByteString.Lazy. This worked just fine with memory usage just increasing a little with each open handle. Also thank you for pointing out the additional packages, I just started learning Haskell and have not yet come across them. – Ben Doerr Oct 01 '12 at 16:42
1

I assume that you are using lazy byte strings (from Data.ByteString.Lazy). There are probably other ways to do this, but one option is to simply use concat :: [ByteString] -> ByteString:

import Control.Monad
import Data.ByteString.Lazy (ByteString)
import qualified Data.ByteString.Lazy as ByteString

readFiles :: [FilePath] -> IO ByteString
readFiles = fmap ByteString.concat . mapM ByteString.readFile

(Note: I don't have time to test the code, but reading the documentation says that this should work)

dflemstr
  • 25,947
  • 5
  • 70
  • 105