1

Let's say I want to save a stream of MyItem into a file (in JSON for example). I would like to stop when the file reach a certain size limit in bytes. I would like to do it in Haskell... streaming is not the issue for me it's more how to get the file size information after each item put into this file...

Nicolas Henin
  • 3,244
  • 2
  • 21
  • 42
  • 1
    Anything wrong with using [hTell](https://hackage.haskell.org/package/base-4.12.0.0/docs/System-IO.html#t:hTell) ? or `hFileSize` from the same System.IO module ? – jpmarinier Apr 10 '20 at 10:30
  • 1
    Can json values be divided and span more than one file? Is the size limit a hard limit, or can it be surpassed to fit in the last `MyItem` in a file? – danidiaz Apr 10 '20 at 10:36
  • @jpmarinier, I don't know what Nicolas needed back then, but one problem is that `hTell` only works for seekable handles (actual files, block devices, etc.). If requirements change and the "file" ends up being a pipe sometimes (or some other sort of handle), then all the code will need to be rewritten. – dfeuer Feb 22 '21 at 17:18
  • @dfeuer - Really, I did not consider such a possibility at the time, given that Nicolas mentioned *a file*. On Linux systems, the underlying `lseek` system call may return `ESPIPE` if for some (implementation) reason the stream does not maintain a proper offset. In that case, I would say the least bad option is probably to have the program logic do the counting, possibly using some object wrapper around the OS level file handle. – jpmarinier Feb 22 '21 at 20:00

1 Answers1

0

I don't think pipes anything for this out of the box, but it's easy to write your own toHandleBounded.

On each step you check the file size and if it's less than the limit you loop, writing another block to the file.

toHandleBounded :: MonadIO m => Integer -> Handle -> Consumer' ByteString m ()
toHandleBounded limit hd = do
  current <- liftIO $ hTell hd
  when (current < limit) $ do
    x <- await
    liftIO $ hPut hd x
    toHandleBounded limit hd

main :: IO ()
main =
  withFile "/dev/urandom" ReadMode $ \hIn ->
  withFile "out.txt" WriteMode $ \hOut ->
    runEffect $ P.fromHandle hIn >-> toHandleBounded (1024 * 500 ) hOut
Masse
  • 4,334
  • 3
  • 30
  • 41