Questions tagged [pipeline]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

In a command line interface or shell, a pipeline uses the pipe operator ("|") to take output from one function or command and input it to another. This is done in a series like "command1 | function1 | command2". For questions related to the pipe operator use the tag.

In computer architecture, a pipeline is a process consisting of a sequence of stages that must be performed in serial order over each element passing the pipe, but may execute in parallel over the elements inside, such that the overall throughput does not depend on the length of the pipe. This is utilized by most CPUs' hardware to process instructions.

A similar technique is also done in software (software-pipelining) in order to optimize the parallelism of a given loop by reordering it to arrange data dependencies in a pipelined manner.

More broadly, "pipeline" is synonymous with "workflow."

See also:

5444 questions
2
votes
2 answers

Assign variables to powershell using C#

I'am tring to call a powershell script and pass through a parameter. I would all so like to pass through a more than one parameter eventually RunspaceConfiguration runspaceConfiguration = RunspaceConfiguration.Create(); Runspace…
Sam Stephenson
  • 5,200
  • 5
  • 27
  • 44
2
votes
1 answer

log mvc pipeline

I'm using asp.net MVC3 and I need to save in a database all the things that user make in my application (where is each click, IP, date, request, client info). where during this processing I can replace the default behavior or inject my own logic? I…
Martin
  • 745
  • 2
  • 7
  • 23
2
votes
1 answer

how to create a csv file dynamically with spider names in scrapy python

I am working on scrapy and scraped a site and fetched all the information Actually I had 3 spiders with different data, I had created these 3 spiders in the same folder with the following structure scrapy.cfg myproject/ __init__.py …
Shiva Krishna Bavandla
  • 25,548
  • 75
  • 193
  • 313
2
votes
1 answer

Does Apache Camel replace Apache commons pipeline

I am implementing a parallelized data processing system that involves a bunch of conversions and filters of data as it moves through multiple stages. I recognize the Apache Commons Pipeline project as a good fit for this requirement, but Apache…
Sajan
  • 21
  • 3
1
vote
2 answers

netty pipelines not getting released from memory

I have a high volume netty server that keeps consuming memory. Using jmap, I've tracked it down to the fact that pipelines just seem to keep growing and growing (along with nio sockets, etc). It is like the socket isn't ever disconnecting. My…
shaggydoug
  • 11
  • 2
1
vote
1 answer

GTKmm + pipe() not working

I have this code: #include #include void send_msg(Gtk::Entry* entry, int* fd); Glib::ustring* receive_msg(int* fd); bool handle_msg(Gtk::Label* lbl, int* fd); int main() { pid_t pid = fork(); int fd[2]; …
Haix64
  • 806
  • 1
  • 13
  • 21
1
vote
1 answer

How we use pipelines item in scrapy

I'm new user of scrapy to crawl my websites.I want to store data crawled into mysql database. myspider.py: class MininovaSpider(CrawlSpider): name = 'myspider' allowed_domains = ['example.com'] start_urls = ['http://www.example.com'] …
user1151311
  • 11
  • 1
  • 3
1
vote
2 answers

grep and find in pipeline. Strange reverse

just find gives me: . ./bla-bla_(11) ./bla-bla_(1) ./rename ./rename~ This find . | grep "*_([0-9]\{1,2\})" gives me empty result. and this find . | grep "([0-9]\{1,2\})_*" gives me ./bla-bla_(11) ./bla-bla_(1) But as you can see underscore and…
Yola
  • 18,496
  • 11
  • 65
  • 106
1
vote
1 answer

Use Result of java XSL transformation as a Stylesheet of a subsequent transformation

I am trying to use the result of an java xsl transform (from XMLSource1.xml and StyleSheet1.xsl) as a stylesheet for another transform (from XMLSource2) and then output the result. I read an interesting article about chaining transformations (also…
1
vote
1 answer

Total execution time of a program with conditional branches in a five-stage pipeline

A CPU has a five-stage pipeline and runs at 1 GHz frequency. Instruction fetch happens in the first stage of the pipeline. A conditional branch instruction computes the target address and evaluates the condition in the third stage of the pipeline.…
user966892
1
vote
1 answer

Executing XPROC

Am very new to XSLT and XPROC stuffs. Now i have written my sample XPROC, As like every beginner, i also started with Hello World. hello.xpl
i2ijeya
  • 15,952
  • 18
  • 63
  • 72
1
vote
1 answer

AppEngine Pipeline Yield - is this standard usage of the yield operator?

From the following page (http://code.google.com/p/appengine-pipeline/wiki/GettingStarted) I have seen the following code in an example of how to use the AppEngine pipeline: class AddOne(pipeline.Pipeline): def run(self, number): return number…
Alexander Marquardt
  • 1,539
  • 15
  • 30
1
vote
1 answer

Azure Synapse Pipeline Get Metadata Last Modified Date on file instead of on folder

I am trying to get the Last Modified date on the files (CSV linked service) in my repository using Get Metadata activity. Field List However, I am getting the same date/time for all of the files... the LastModified date on the folder instead of the…
1
vote
2 answers

Powershell: Cmdlet to access sub properties in the pipeline

In powershell (7.3.x), I'm currently reading a json file (or an xml file) and I want to access a nested property within the pipeline without having to wrap stuff in parentheses. Let me explain what I've found, and what I'm actually looking…
Jiehong
  • 786
  • 1
  • 7
  • 16
1
vote
1 answer

Why do I get "Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED))" error when opening Monogame pipeline?

When I try to open MonoGame pipeline (MGCB Editor), I get an error that says either "Catastrophic failure (Exception from HRESULT: 0x8000FFFF (E_UNEXPECTED))" or "The Extender Provider failed to return an Extender for this object". The editor still…
Absol
  • 21
  • 2