2

Background

I have a long stream of large files whose contents I would like to stream in constant memory.

I'm using the Haskell Pipes library to model this stream with the following nested type:

Producer (FilePath, Producer ByteString IO ()) IO ()

In other words, the outer producer iterates over files, and the inner producer iterates over file chunks.

Problem

For accounting purposes, I'd like to temporarily flatten this nested stream into something with the following type:

Producer FileStreamEvent IO ()

where:

type FileStreamEvent = NestedStreamEvent FilePath ByteString

data NestedStreamEvent a b = NestedStreamOpen  a
                           | NestedStreamChunk b
                           | NestedStreamClose a

It seems easy to write a function to perform this transformation:

flattenNestedStream :: Monad m
                    => Producer (a, Producer b m r) m r
                    -> Producer (NestedStreamEvent a b) m r
flattenNestedStream abs = for abs $ \(a, bs) -> do
    yield                $ NestedStreamOpen  a
    for bs $ \b -> yield $ NestedStreamChunk b
    yield                $ NestedStreamClose a

However, I'm not sure how to write the inverse transformation:

nestFlattenedStream :: Monad m
                    => Producer (NestedStreamEvent a b) m r
                    -> Producer (a, Producer b m r) m r

such that:

(nestFlattenedStream . flattenNestedStream) producer == producer

Can the function nestFlattenedStream be defined?

jsk
  • 285
  • 1
  • 7
  • 1
    Can you describe more of what you are doing? Perhaps there is a solution which doesn't require you to re-nest the flattened stream. – ErikR Oct 08 '15 at 13:00
  • I think, like @ErikR , a `Producer (FilePath, Producer ByteString IO ()) IO r` definitely shouldn't arise. The things piped and produced should generally be ordinary values. A `Producer FilePath (Producer ByteString IO) r` could make sense, but is a little exotic. (Something like could be necessary though.) Can you say more about how you got this `Producer (FilePath, Producer a m r) m r` and where you are going with it? – Michael Oct 08 '15 at 23:04

0 Answers0