Stream from `Managed` pipe

Question

I'm using Control.Monad.Managed to wrap pipes-http's withHTTP in a continuation transformer, but I also want to stream the HTTP response out afterwards:

import Pipes
import Pipes.ByteString
import Pipes.HTTP
import Control.Applicative
import Control.Monad
import Control.Monad.Managed
import qualified Data.ByteString.Char8       as BC

type RespBody = BC.ByteString

managedHTTP :: Request -> Manager -> Managed (Response (Producer RespBody IO ()))
managedHTTP r m = managed (withHTTP r m)

fromHTTP :: Request -> Manager -> Producer RespBody IO ()
fromHTTP request manager = join $ liftIO $ fmap (hoist liftIO) $ with response return
  where response = responseBody <$> managedHTTP request manager

Here fromHTTP is meant to stream out the response body. If the above is run with:

main = do
  request <- parseUrl "http://en.wikipedia.org/"
  manager <- newManager defaultManagerSettings
  runEffect $ fromHTTP request manager >-> stdout

I get recv: invalid argument (Bad file descriptor) after the first chunk because http-client's responseClose already closed my response body.

How should I go about streaming HTTP responses with pipes if I want to expose a non-with style interface?

Thanks & apologies if I'm seriously misunderstanding continuations and pipes.

Michael · Accepted Answer · 2014-09-09T18:01:24.927

1

Your definition of fromHTTP is a little hard to follow, but it clearly has you exiting the managed environment before you actually start using the producer. Something like this might be closer to what you intend:

import Pipes
import Pipes.ByteString
import Pipes.HTTP
import Control.Applicative
import Control.Monad
import Control.Monad.Managed
import qualified Data.ByteString.Char8       as BC

type RespBody = BC.ByteString

managedHTTP :: Request -> Manager -> Managed (Response (Producer RespBody IO ()))
managedHTTP r m = managed (withHTTP r m)

managedManager :: ManagerSettings -> Managed Manager
managedManager settings = managed (withManager settings)

main = runManaged $ 
  do request <- liftIO $ parseUrl "http://en.wikipedia.org/" 
     manager <-  managedManager defaultManagerSettings
     producer <- managedHTTP request manager
     liftIO $ runEffect $ responseBody producer >-> stdout

or just scrap the definitions and write

main = runManaged $ 
  do request  <- liftIO $ parseUrl "http://en.wikipedia.org/" 
     manager  <- managed (withManager defaultManagerSettings)
     producer <- managed (withHTTP request manager)
     liftIO $ runEffect $ responseBody producer >-> stdout

Note that you can make the producer run in the Managed monad as well:

main = runManaged $ 
  do request  <- liftIO $ parseUrl "http://en.wikipedia.org/" 
     manager  <- managed (withManager defaultManagerSettings)
     producer <- managed (withHTTP request manager)
     runEffect $ hoist liftIO (responseBody producer) >-> stdout

edited Sep 09 '14 at 18:01

answered Sep 09 '14 at 15:30

Michael

2,889
17
16

I originally didn't want ``fromHTTP`` to be ``Managed (Producer ...)`` because I have a big transformer stack inside the Producer, and I was assuming the base monad is IO. I was able to fix this problem with ``Control.Monad.Base`` (making ``Proxy`` an instance of ``MonadBase``) – tranma Sep 09 '14 at 16:56
This solution works if you can afford to have ``fromHTTP`` in a ``Managed`` (instead of over it) though. – tranma Sep 09 '14 at 16:59
Oh, I see. It is a little annoying that Managed is so restricted, but I suppose it helps make the package intelligible. There is also the expedient of directly using `Codensity` as is described here http://hackage.haskell.org/package/pipes-4.1.2/docs/Pipes-Tutorial.html#g:10 (for another reason), using the `Condensity` constructor as `managed` and `lowerCodensity` as `runManaged`. Or maybe I'm missing something. – Michael Sep 09 '14 at 17:36
I see, the trouble is coming from the types of `withHTTP` and `withManager` so Codensity wouldn't work immediately. Is there an efficiency problem with using `fromHTTP` "in a `Managed` instead of over it"? – Michael Sep 09 '14 at 18:03
I'm not sure if there would be an efficiency problem with having ``fromHTTP :: Managed (Producer x m r)`` as opposed to: ``fromHTTP :: MonadBase Managed m => Producer x m r``. My motivation was that I'd like to use ``fromHTTP`` as a data source inside other pipes and ``ListT``s, so I thought it's more composeable this way. – tranma Sep 10 '14 at 03:09
In retrospect, perhaps it wasn't the best idea to have a big stack underneath the producers. I ended up with pieces like ``foo :: (ReaderT Env `In` m, MonadBase Managed m, ...) => Pipe x y m r`` and ``run`` functions to peel off each layer underneath the pipe. I'm not sure how best to compose streaming code like this though. – tranma Sep 10 '14 at 03:13

Stream from `Managed` pipe

1 Answers1