`pipes` is a group of libraries written in Haskell to provide safe, functional, and strict I/O.
Questions tagged [haskell-pipes]
139 questions
6
votes
2 answers
Folding a subset of a stream using pipes 4.0
I'm trying to understand pipes 4.0, and want to convert some conduit code. Suppose I have a stream of Ints, and I'd like to skip the first five, then get the sum of the following 5. Using plain lists, this would be:
sum . take 5 . drop 5
In…
Michael Snoyman
- 31,100
- 3
- 48
- 77
6
votes
2 answers
How can I have a pipe with multiple communication types?
Say I have this code:
import Control.Monad.State hiding (StateT)
import Control.Proxy
server :: (Proxy p, Monad m) => Int -> Server p Int Bool (StateT Int m) ()
server = runIdentityK loop
where loop arg = do
currMax <- lift get
…
ajp
- 1,723
- 14
- 22
5
votes
0 answers
Haskell pipes: how to write bytestring to file
I have the following code that should create a pipe Consumer for [Word8] and output it to a file.
import Pipes
import qualified Pipes.Binary as PB
import qualified Pipes.ByteString as PBS
import qualified System.IO as SIO
save :: String ->…
JS0
- 173
- 4
5
votes
1 answer
How would I pipe with a timeout that resets with each incoming?
The withTimeout function is suppose to pipe ConsoleEvent with a CeTimeout sent every s :: Int seconds if nothing has been received. Instead it fails to send the CeTimeout events at the appropriate times. One CeTimeout event is replaced for other…
Vanson Samuel
- 1,953
- 2
- 17
- 30
5
votes
1 answer
Optimizing Memory in Haskell, pipes, attoparsec, and containers
I'm trying to further optimize my pipes-attoparsec parser and storage, but having trouble getting memory usage any lower.
Given account-parser.hs
{-# LANGUAGE RankNTypes #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE NoImplicitPrelude…
Adam Flott
- 487
- 3
- 8
5
votes
3 answers
Efficient streaming and manipulation of a byte stream in Haskell
While writing a deserialiser for a large ()* encoded binary file I got stuck with the various Haskell produce-transform-consume libraries. So far I'm aware of four streaming libraries:
Data.Conduit: Widely used, has very careful…
mcmayer
- 1,931
- 12
- 22
5
votes
2 answers
Haskell Pipes - get return value of last Proxy in pipeline
Let's say I have two Proxy in Haskell Pipes. They represent external system processes.
produce :: MonadIO m => Producer ByteString m ExitCode
consume :: MonadIO m => Consumer ByteString m ExitCode
So I hook them into an Effect, like this:
effect…
massysett
- 1,100
- 6
- 13
5
votes
1 answer
How to detect end of input with pipes
I'm trying to read a group of up to 50 items from a pipe and process them in an IO action all at once. (The use case for this is I'm trying to insert data into a database and I want to do an entire batch inside one transaction because it is vastly…
Gareth Charnock
- 1,166
- 7
- 17
5
votes
1 answer
Concurrency considerations between pipes and non-pipes code
I'm in the process of wrapping a C library for some encoding in a pipes interface, but I've hit upon some design decisions that need to be made.
After the C library is set up, we hold on to an encoder context. With this, we can either encode, or…
gspr
- 11,144
- 3
- 41
- 74
5
votes
2 answers
Converting normal attoparsec parser code to conduit/pipe based
I have written a following parsing code using attoparsec:
data Test = Test {
a :: Int,
b :: Int
} deriving (Show)
testParser :: Parser Test
testParser = do
a <- decimal
tab
b <- decimal
return $ Test a b
tParser :: Parser…
Sibi
- 47,472
- 16
- 95
- 163
5
votes
3 answers
Limiting pipes based on time?
Is it possible to create pipes that get all values that have been sent downstream in a certain time period? I'm implementing a server where the protocol allows me to concatenate outgoing packets and compress them together, so I'd like to…
user3261399
- 347
- 1
- 5
5
votes
1 answer
Is this a sane architecture for a multi-user network server? (How much overhead does pipes-concurrency introduce?)
I currently have it so that there is one thread handling the accept loop, one main thread for doing all the stateful logic stuff, and then 2 threads per connected client. One client thread is handling the input pipeline and using pipes-concurrency…
user3261399
- 347
- 1
- 5
5
votes
1 answer
can I make StateP from Control.Proxy an instance of MonadState?
I am changing some code that used to run inside a StateT monad to run inside of StateP from Control.Proxy. However, some of my code (e.g. the %= operator from Control.Lens) requires a MonadState instance. Is it safe/correct for me to just add such…
ajp
- 1,723
- 14
- 22
5
votes
2 answers
"Sub-parsers" in pipes-attoparsec
I'm trying to parse binary data using pipes-attoparsec in Haskell. The reason pipes (proxies) are involved is to interleave reading with parsing to avoid high memory use for large files. Many binary formats are based on blocks (or chunks), and their…
absence
- 821
- 1
- 8
- 13
4
votes
1 answer
How do I know where in my Haskell Program an Exception can be Thrown?
I'm working on a 1-10k line personal Haskell project, and I'm using various IO-libraries such as Network.HTTP.Conduit inside workers spawned with Pipes.Concurrent.
I just realized that some of these IO libraries are throwing exceptions in edge…
Craig
- 255
- 1
- 6