Questions tagged [pipeline]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

In a command line interface or shell, a pipeline uses the pipe operator ("|") to take output from one function or command and input it to another. This is done in a series like "command1 | function1 | command2". For questions related to the pipe operator use the tag.

In computer architecture, a pipeline is a process consisting of a sequence of stages that must be performed in serial order over each element passing the pipe, but may execute in parallel over the elements inside, such that the overall throughput does not depend on the length of the pipe. This is utilized by most CPUs' hardware to process instructions.

A similar technique is also done in software (software-pipelining) in order to optimize the parallelism of a given loop by reordering it to arrange data dependencies in a pipelined manner.

More broadly, "pipeline" is synonymous with "workflow."

See also:

5444 questions
15
votes
1 answer

In what conditions does powershell unroll items in the pipeline?

Consider the following: function OutputArray{ $l = @(,(10,20)) $l } (OutputArray) -is [collections.ienumerable] # C:\ PS> True (OutputArray).Count # C:\ PS> 2 $l is "unrolled" when it enters the pipeline. This answer states that…
alx9r
  • 3,675
  • 4
  • 26
  • 55
15
votes
1 answer

Proper way to unpipe a streams2 pipeline and empty it (not just flush)

Premise I'm trying to find the correct way to prematurely terminate a series of piped streams (pipeline) in Node.js: sometimes I want to gracefully abort the stream before it has finished. Specifically I'm dealing with mostly objectMode: true and…
zamnuts
  • 9,492
  • 3
  • 39
  • 46
15
votes
1 answer

Issuing multiple requests before getting response

I'm having trouble understanding how HTTP works when multiple requests are send parallely (before getting a response). There are two cases: 1) With Connection: Keep-Alive. According to HTTP spec: A client that supports persistent connections MAY…
freakish
  • 54,167
  • 9
  • 132
  • 169
15
votes
5 answers

R Pipelining functions

Is there a way to write pipelined functions in R where the result of one function passes immediately into the next? I'm coming from F# and really appreciated this ability but have not found how to do it in R. It should be simple but I can't find…
Matthew Crews
  • 4,105
  • 7
  • 33
  • 57
14
votes
2 answers

closing stdout of piped python subprocess

Here is what I can read in the python subprocess module documentation: Replacing shell pipeline output=`dmesg | grep hda` ==> p1 = Popen(["dmesg"], stdout=PIPE) p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE) …
user941068
14
votes
4 answers

How do I use output from awk in another command?

So I need to convert a date to a different format. With a bash pipeline, I'm taking the date from the last console login, and pulling the relevant bits out with awk, like so: last $USER | grep console | head -1 | awk '{print $4, $5}' Which outputs:…
snk
14
votes
1 answer

Service Fabric Reliable Services Pipeline design

I need to implement pipeline if Service Fabric's Reliable Services, and I need some guidelines about what of these approaches is preferable from the viewpoint of reliability simplicity and simple good design:
AsValeO
  • 2,859
  • 3
  • 27
  • 64
14
votes
4 answers

How to pipeline in node.js to redis?

I have lot's of data to insert (SET \ INCR) to redis DB, so I'm looking for pipeline \ mass insertion through node.js. I couldn't find any good example/ API for doing so in node.js, so any help would be great!
Aviram Netanel
  • 12,633
  • 9
  • 45
  • 69
13
votes
1 answer

How to use rule in gitlab-ci

I'm trying to build a job that can be conditionally executed depends on whether the files or subdirectories in WebClient is modified in develop branch, using rules. If there are changes found in the develop branch only, then a pipeline will be…
13
votes
1 answer

Sklearn components in pipeline is not fitted even if the whole pipeline is?

I'm trying to single out a component/transformer from a fitted pipeline to inspect it's behavior. However, when I retrieved the component, the component is showed as unfitted, but using the pipeline as a whole works without problem. This suggest the…
Alex Ramses
  • 538
  • 3
  • 19
13
votes
2 answers

Gitlab CI in multiple platforms simultaneously

I have a C++ project that is compiled and packaged for multiple OS (Linux, Windows, MacOS) as well as multiple CPU architectures (i386, x86_64, arm, Aarch64) For this I'm using Jenkins to grab the source code and run the build script in parallel on…
aprad046
  • 636
  • 7
  • 12
13
votes
5 answers

C++ Iterator Pipelining Designs

Suppose we want to apply a series of transformations, int f1(int), int f2(int), int f3(int), to a list of objects. A naive way would be SourceContainer source; TempContainer1 temp1; transform(source.begin(), source.end(), back_inserter(temp1),…
kirakun
  • 2,770
  • 1
  • 25
  • 41
13
votes
6 answers

name 'DataFrameSelector' is not defined

I am currently reading the "Hands-On Machine Learning with Scikit-Learn & TensorFlow". I get an error when I am trying to recreate the Transformation Pipelines code. How can I fix this? Code: from sklearn.pipeline import Pipeline from…
Isaac A
  • 543
  • 1
  • 6
  • 18
13
votes
2 answers

What happens when reading into a variable in a pipeline?

echo hello | read str echo $str This read is executed after the pipeline, which means that the output of the echo gets read into str - but because it is after a pipe, the contents of str are now in a subshell that cannot be read by the parent…
capser
  • 2,442
  • 5
  • 42
  • 74
13
votes
3 answers

Create Jenkins Docker Image with pre configured jobs

I have created a bunch of Local deployment pipeline jobs, these jobs do things like remove an existing container, build a service locally, build a docker image, run the container - etc. These are not CI/CD jobs, just small pipelines for deploying…
Tim Jarvis
  • 18,465
  • 9
  • 55
  • 92