`pipes` is a group of libraries written in Haskell to provide safe, functional, and strict I/O.
Questions tagged [haskell-pipes]
139 questions
7
votes
1 answer
what is the relationship between Haskell's FreeT and Coroutine type
In the "Coroutine Pipelines" article in Monad.Reader Issue 19, the author defines a generic Coroutine type:
newtype Coroutine f m a = Coroutine
{ resume :: m (Either (f (Coroutine f m a)) a)
}
I noticed that this type is very similar to the…
illabout
- 3,517
- 1
- 18
- 39
7
votes
1 answer
How to turn a pull based pipe into a push based one?
By default pipes are pull based. This is due to the operator >-> which is implemented via +>> which is the pointful bind operator for his pull category. My understanding is that this means that if you have code like producer >-> consumer, the…
David McHealy
- 2,471
- 18
- 34
7
votes
1 answer
How come `Effect` seals only two in-flows, instead of all the flows?
Here is the diagram of Effect provided in the official tutorial of pipes package.
type Effect = Proxy X () () X
Upstream | Downstream
+---------+
| |
X <== <== ()
| |
() ==> ==> X
| | |
…
xzhu
- 5,675
- 4
- 32
- 52
7
votes
1 answer
Separation of data loading/unloading and processing logic
Sometimes it is necessary to perform some complex routines in order to retrieve or save data, which is being processed. In this case one wants to separate data generation and data processing logic. The common way is to use iteratee-like…
user3974391
- 701
- 4
- 14
7
votes
1 answer
Simple program using Pipes hangs
I have the following program, which produces no output when run with runhaskell Toy.hs, and instead hangs indefinitely. By my understanding, the program should print "hi" and then exit. I would appreciate an answer and/or advice about how to debug…
ajp
- 1,723
- 14
- 22
6
votes
2 answers
Pipes Tutorial: ListT example
I'm trying to make sense of one of the examples presented at the pipes tutorial concerning ListT:
import Pipes
import qualified Pipes.Prelude as P
input :: Producer String IO ()
input = P.stdinLn >-> P.takeWhile (/= "quit")
name :: ListT IO…
Damian Nadales
- 4,907
- 1
- 21
- 34
6
votes
2 answers
Read large lines in huge file without buffering
I was wondering if there's an easy way to get lines one at a time out of a file without eventually loading the whole file in memory. I'd like to do a fold over the lines with an attoparsec parser. I tried using Data.Text.Lazy.IO with hGetLine and…
Charles Durham
- 1,707
- 11
- 17
6
votes
1 answer
Why can't Conduit and Pipe have an Arrow instance?
There is an archived thread on reddit which says essentially conduit/pipes cannot be arrows b/c arrows need to be synchronous. The thread is linked here https://www.reddit.com/r/haskell/comments/rq1q5/conduitssinks_and_refactoring_arrows/
I fail to…
user2812201
- 437
- 3
- 7
6
votes
1 answer
How would you traverse a directory and do some function on all files and combine the output in a memory efficient manner?
Setting
I need to traverse a directory over 100+ .txt files, open every one and do some function on each, then combine the results. These files are huge, on the order of 10GB. Some common operation in psuedocode might be:
foldr concatFile mempty $…
xiaolingxiao
- 4,793
- 5
- 41
- 88
6
votes
1 answer
Signalling to downstream that upstream is exhausted
Question
Using the Haskell pipes library, I'm trying to define a Pipe with the following type:
signalExhausted :: Monad m => Pipe a (Value a) m r
where the Value data type is defined by:
data Value a = Value a | Exhausted
The pipe should obey the…
jsk
- 285
- 1
- 7
6
votes
1 answer
Using Pipes to read and write binary data in Haskell
I am trying to read and write very many ints in constant memory. I have figured out how to write the ints to memory but have not figured out how to read them back.
import Control.Lens (zoom)
import System.IO (IOMode(..), withFile)
import…
Justin Raymond
- 3,413
- 2
- 19
- 28
6
votes
1 answer
why pipes defines inner functions
I'm looking at the pipes library source code and for instance in the Core module I don't understand why the author is all over the place using the pattern of defining functions like that:
runEffect = go
where
go p = ...
Or:
pull = go
where
…
Emmanuel Touzery
- 9,008
- 3
- 65
- 81
6
votes
2 answers
using haskell pipes-bytestring to iterate a file by line
I am using the pipes library and need to convert a ByteString stream to a stream of lines (i.e. String), using ASCII encoding. I am aware that there are other libraries (Pipes.Text and Pipes.Prelude) that perhaps let me yield lines from a text file…
Stephan
- 746
- 5
- 14
6
votes
1 answer
Space explosion when folding over Producers/Parsers in Haskell
Supposing I have a module like this:
module Explosion where
import Pipes.Parse (foldAll, Parser, Producer)
import Pipes.ByteString (ByteString, fromLazy)
import Pipes.Aeson (DecodingError)
import Pipes.Aeson.Unchecked (decoded)
import Data.List…
immutablestate
- 287
- 1
- 9
6
votes
1 answer
How to use State with Pipes?
I have a function with a type Map Int String -> Proxy () a () Void IO b. Right now it awaits, does whatever with the value it got, and then re-calls itself. I'd like to change it to use State (Map Int String) instead of having that passed as an…
user3261399
- 347
- 1
- 5