Questions tagged [pipeline]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

In a command line interface or shell, a pipeline uses the pipe operator ("|") to take output from one function or command and input it to another. This is done in a series like "command1 | function1 | command2". For questions related to the pipe operator use the tag.

In computer architecture, a pipeline is a process consisting of a sequence of stages that must be performed in serial order over each element passing the pipe, but may execute in parallel over the elements inside, such that the overall throughput does not depend on the length of the pipe. This is utilized by most CPUs' hardware to process instructions.

A similar technique is also done in software (software-pipelining) in order to optimize the parallelism of a given loop by reordering it to arrange data dependencies in a pipelined manner.

More broadly, "pipeline" is synonymous with "workflow."

See also:

5444 questions
2
votes
1 answer

Consume data from pipeline

How to consume data from pipeline when writing cmdlets in C#? For example I have two classes: This one produces data: [Cmdlet(VerbsCommon.Get, "Numbers")] public class GetNumbers : Cmdlet { protected override void ProcessRecord() { …
Vitaliy
  • 702
  • 6
  • 19
2
votes
2 answers

What happens when pipe char directly followed by the redirection in Bash?

I accidentally ran the following scripts in Bash: $ ls -l | > ../test.txt And I got an empty test.txt. What happened?
cqzlxl
  • 23
  • 2
2
votes
1 answer

Store scrapy items to process after spider completes

I'm working on writing a scrapy pipeline that will call a function to clear our cdn's edge servers of the scraped urls. I figured out how to store the list of visited urls easily enough, but the issue is knowing when the crawler is done. The cdn's…
avaleske
  • 1,793
  • 5
  • 16
  • 26
2
votes
0 answers

How does the graphics pipeline use shared memory?

In GPGPU algorithms, using shared or local memory is important to achieve high performance. Since GPUs are made to run the graphics pipeline, the shared memory has to have some important use, but I can not figure out what. Someone who has read the…
nulvinge
  • 1,600
  • 8
  • 17
2
votes
1 answer

Go lang closure pipeline deadlock

I'm working on a data import job using the Go language, I want to write each step as a closure, and use channels for communication, that is, each step is concurrent. The problem can be defined by the following structure. Get Widgets from data…
Lee Hambley
  • 6,270
  • 5
  • 49
  • 81
2
votes
2 answers

Is a pipeline with a changing data type architecturally sound?

I'm working on the architecture for what is essentially a document parsing and analysis framework. Given the lines of the document, the framework will ultimately produce a large object (call it Document) representing the document. Early filters in…
Chris Laplante
  • 29,338
  • 17
  • 103
  • 134
2
votes
2 answers

IIS 7 Applications and asp.net - newbie question

Just getting started on a project to migrate from win 2003 iis6 to win 2008 / IIS7, and after reading the MS documentation and also various articles I am a little confused, as it states a site needs to have one or more applications. However I have…
sjb101
  • 131
  • 1
  • 6
2
votes
1 answer

count number of words and lines from stdin

How can i write a program using STANDARD UNIX UTILITIES that will read data from standard input one character at a time and out the results to standard output. I know that it runs similar to what a C program in this case. I was told that this could…
user1832605
  • 93
  • 1
  • 3
  • 10
2
votes
3 answers

R Pipelining with Anonymous Functions

I have a question which is an extension of another question. I am wanting to be able to pipeline anonymous functions. In the previous question the answer to pipeline defined functions was to create a pipeline operator "%|>%" and to define it this…
Matthew Crews
  • 4,105
  • 7
  • 33
  • 57
2
votes
1 answer

In Biztalk, how do I split an envelope with an extra element?

I'm trying to split an incoming message in the following form: 123 ... ... ... I've a pipeline with a XML disassembler which takes the Items schema and…
James Curran
  • 101,701
  • 37
  • 181
  • 258
2
votes
0 answers

Implementing pipelining in c. What would be the best way to do that? (Own shell in linux)

Possible Duplicate: Implementing pipelining in c. What would be the best way to do that? (Own linux shell) I can't think of any way to implement pipelining in c that would actually work. That's why I've decided to write in here. I have to say,…
Patryk
  • 3,042
  • 11
  • 41
  • 83
2
votes
0 answers

Video and Audio player for AVI using Python and pipelines not working

I have been looking for the method to use avidemux, pipelines and python to display the video and audio from a single avi video file. I can do it using gst-launch but i wanted to implement it in code. Right now it sais that all the elements are…
Raxus
  • 21
  • 2
2
votes
2 answers

Finding a File With Specific Permissions

I need to write a command pipeline that will show all non-hidden files that have read permissions for all users. I dont know why this wouldn't work: find * -perm a=r -print I get no output and am not sure where I am going wrong. Please Help.
Mack Gray
  • 147
  • 1
  • 1
  • 6
2
votes
1 answer

Pass multiple variables to PowerShell from C#

I'm invoking a PowerShell script from C#, I'm using a hashtable to pass through parameters however when it comes to invoking the PowerShell script I get A positional parameter cannot be found that accepts argument The PowerShell script has two…
Sam Stephenson
  • 5,200
  • 5
  • 27
  • 44
2
votes
2 answers

netty ChannelPipeline addLast

I am learning netty and there is a following code from example ChannelPipeline pipeline = pipeline(); // Enable stream compression (you can remove these two if unnecessary) pipeline.addLast("deflater", new…
Nikolay Kuznetsov
  • 9,467
  • 12
  • 55
  • 101