Why reading of file takes memory for the whole file?

Question

I'm doing this:

import qualified Data.ByteString.Lazy.Char8 as BS

main :: IO ()
main = do
    let filename = "sample.txt"    
    text <- BS.readFile filename
    let res = BS.take 1000 $ text
    print res

When I run this with profiling it gave me:

162,048 bytes allocated in the heap
2,472 bytes copied during GC
59,688 bytes maximum residency (1 sample(s))
22,232 bytes maximum slop
156 MB total memory in use (0 MB lost due to fragmentation)

File that I read is about 50Kbytes. Why it takes 60Kbytes memory (maximum residency)? I have tried with String and Lazy text, too. It's the same picture. I think Haskell in some way is reading whole file into memory or just allocate as much memory as is the file long. How could I prevent this? I would like to read from it only N bytes and don't want to waste so much memory.

with `runghc` take 57M but compiling `ghc -O3 t.hs` take only ` 61,816 bytes maximum residency` (max real memory usage `2M`, my sample.txt is `84M`) — josejuan, Jan 28 '14 at 10:50
@josejuan Hmm, I can't reproduce the behavior you see with `runghc`. It takes ~60K for me on large file. And I don't see how it depends on optimization... — Yuras, Jan 28 '14 at 11:02

score 10 · Accepted Answer · answered Jan 28 '14 at 10:59

Your file's size is 50K, but readFile reads file by 32k chunks. So you have half of the file in memory. Except that, some memory is used for runtime system, gc, intermediate lazy string in print, etc. So 60K max residency is ok.

You can try with bigger file, e.g. 100M. You'll see that max residency will not grow.

If the default chunk size is not ok for you, then try hGetSome

Why reading of file takes memory for the whole file?

1 Answers1