Stack Overflow
Stack Overflow

Questions tagged [conduit]

conduit is a Haskell library for composable processing of streaming data.

conduit is a Haskell library for composable processing of streaming data. It was developed as an alternative to iteratee implementations such as enumerator and is now used by several popular packages, including WAI and http-conduit (formerly http-enumerator).

255 questions
10
votes
2 answers

What's the real benefit of conduit's upstream type parameter?

I'm trying to understand the differences between different implementations of the concept of pipes. One of the differences between conduit and pipes is how they fuse pipes together. Conduit has (>+>) :: Monad m => Pipe l a b r0 m r1 -> Pipe…
Petr
  • 62,528
  • 13
  • 153
  • 317
9
votes
1 answer

Rechunk a conduit into larger chunks using combinators

I am trying to construct a Conduit that receives as input ByteStrings (of around 1kb per chunk in size) and produces as output concatenated ByteStrings of 512kb chunks. This seems like it should be simple to do, but I'm having a lot of trouble,…
Rehno Lindeque
  • 4,236
  • 2
  • 23
  • 31
9
votes
1 answer

How do I make a "branched" Conduit?

I want the same data to be split in two "branches" to be processed separately, then "joined"... +----------+ +---------+ -->| doublber |--- +--------+ +--------+ | |-- +----------+ …
Vi.
  • 37,014
  • 18
  • 93
  • 148
9
votes
2 answers

Is it safe to reuse a conduit?

Is it safe to perform multiple actions using the same conduit value? Something like do let sink = sinkSocket sock something $$ sink somethingElse $$ sink I recall that in the early versions of conduit there were some dirty hacks that made…
Roman Cheplyaka
  • 37,738
  • 7
  • 72
  • 121
9
votes
2 answers

Fusing conduits with multiple inputs

I am trying to create a conduit that can consume multiple input streams. I need to be able to await on one or the other of the input streams in no particular order (e.g., not alternating) making zip useless. There is nothing parallel or…
Benson
  • 121
  • 5
9
votes
1 answer

How to use the conduit drop function in a pipeline?

I have a simple task - read a bunch of lines out of a file and do something with each one of them. Except the first one - which are some headings to be ignored. So I thought I'd try out conduits. printFile src = runResourceT $ CB.sourceFile src =$=…
Oliver
  • 2,361
  • 2
  • 17
  • 12
8
votes
1 answer

Ways to improve performance for line-based conduits

I use haskell for line-based data processing, i.e. tasks where you can apply sed, awk and similar tools. As a trivial example, let's prepend 000 to every line from standard input. I have three alternative ways to do the task: Lazy IO with lazy…
modular
  • 1,099
  • 9
  • 22
8
votes
2 answers

Conduit - Multiple output file within the pipeline

I'm writing a programme where an input file is split into multiple files (Shamir's Secret Sharing Scheme). Here's the pipeline I'm imagining: source: use Conduit.Binary.sourceFile to read from the input conduit: Takes a ByteString, produces…
Jacob Wang
  • 4,411
  • 5
  • 29
  • 43
8
votes
4 answers

conduit and sockets: allow multiple connections

Here's some code that implements a small receiving server using conduit, network-conduit, and stm-conduit. It receives data on a socket and then streams it through an STM-channel to the main thread. import Control.Concurrent (forkIO) import…
Impredicative
  • 5,039
  • 1
  • 17
  • 43
8
votes
1 answer

Conduit Broadcast

A view days ago, I asked this question. Now I need a pure single threaded version of this function: To repeat, I need a function that sends each received value to each sink and collects their results. The type signature of the function should be…
SvenK
  • 2,584
  • 2
  • 21
  • 24
8
votes
1 answer

Conduit: Multiple Stream Consumers

I write a program which counts the frequencies of NGrams in a corpus. I already have a function that consumes a stream of tokens and produces NGrams of one single order: ngram :: Monad m => Int -> Conduit t m [t] trigrams = ngram 3 countFreq :: (Ord…
SvenK
  • 2,584
  • 2
  • 21
  • 24
8
votes
2 answers

Haskell http-conduit web-scraping daemon crashes with out of memory error

I've written a daemon in Haskell that scrapes information from a webpage every 5 minutes. The daemon originally ran fine for about 50 minutes, but then it unexpectedly died with out of memory (requested 1048576 bytes). Every time I ran it it died…
nejstastnejsistene
  • 111
  • 1
  • 6
7
votes
1 answer

How can I turn a Sink into a Conduit?

I'm trying to write a Conduit using an attoparsec parser. Specifically, given parseOne :: Parser T, I'd like to construct a Conduit ByteString m T that repeatedly applies the parser to the input and streams the results. attoparsec-conduit offers…
ehird
  • 40,602
  • 3
  • 180
  • 182
7
votes
1 answer

what is the relationship between Haskell's FreeT and Coroutine type

In the "Coroutine Pipelines" article in Monad.Reader Issue 19, the author defines a generic Coroutine type: newtype Coroutine f m a = Coroutine { resume :: m (Either (f (Coroutine f m a)) a) } I noticed that this type is very similar to the…
illabout
  • 3,517
  • 1
  • 18
  • 39
7
votes
1 answer

Is there identity conduit?

The title says it all. I've seen that some people apparently use Data.Conduit.List.map id as identity conduit, but is this the recommended way to stream data unchanged?
Mark Karpov
  • 7,499
  • 2
  • 27
  • 62
1
2
3
…
16 17