Questions tagged [pipeline]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

In a command line interface or shell, a pipeline uses the pipe operator ("|") to take output from one function or command and input it to another. This is done in a series like "command1 | function1 | command2". For questions related to the pipe operator use the tag.

In computer architecture, a pipeline is a process consisting of a sequence of stages that must be performed in serial order over each element passing the pipe, but may execute in parallel over the elements inside, such that the overall throughput does not depend on the length of the pipe. This is utilized by most CPUs' hardware to process instructions.

A similar technique is also done in software (software-pipelining) in order to optimize the parallelism of a given loop by reordering it to arrange data dependencies in a pipelined manner.

More broadly, "pipeline" is synonymous with "workflow."

See also:

5444 questions
17
votes
3 answers

Should I create pipeline to save files with scrapy?

I need to save a file (.pdf) but I'm unsure how to do it. I need to save .pdfs and store them in such a way that they are organized in a directories much like they are stored on the site I'm scraping them off. From what I can gather I need to make…
John Lotacs
  • 1,184
  • 4
  • 20
  • 34
17
votes
3 answers

How to prevent a step failing in Bitbucket Pipelines?

I am running all my test cases and some of them get fail sometimes, pipeline detects it and fail the step and build. this blocks the next step to be executed (zip the report folder). I want to send that zip file as an email attachment. Here is my …
Shubhendu Pandey
  • 175
  • 1
  • 1
  • 6
17
votes
3 answers

how to use "if" statements inside pipeline

I'm trying to use if inside a pipeline. I know that there is where (alias ?) filter, but what if I want activate a filter only if a certain condition is satisfied? I mean, for example: get-something | ? {$_.someone -eq 'somespecific'} |…
Emiliano Poggi
  • 24,390
  • 8
  • 55
  • 67
17
votes
1 answer

Sklearn Pipeline - How to inherit get_params in custom Transformer (not Estimator)

I have a pipeline in scikit-learn that uses a custom transformer I define like below: class MyPipelineTransformer(TransformerMixin): which defines functions __init__, fit() and transform() However, when I use the pipeline inside…
Max Power
  • 8,265
  • 13
  • 50
  • 91
16
votes
1 answer

The realationship between window size and sequence number

The question is : We have a transport protocoll that uses pipelining and use a 8-bit long sequence number (0 to 255) What is the maximum window size sender can use ? (How many packets the sender can send out on the net before it muse wait for an…
Jan
  • 175
  • 1
  • 1
  • 5
16
votes
4 answers

Sklearn pass fit() parameters to xgboost in pipeline

Similar to How to pass a parameter to only one part of a pipeline object in scikit learn? I want to pass parameters to only one part of a pipeline. Usually, it should work fine like: estimator = XGBClassifier() pipeline = Pipeline([ ('clf',…
Georg Heiler
  • 16,916
  • 36
  • 162
  • 292
16
votes
4 answers

R: combine several gsub() function in a pipe

To clean some messy data I would like to start using pipes %>%, but I fail to get the R code working if gsub() is not at the beginning of the pipe, should occur late (Note: this question is not concerned with proper import, but with data…
user2006697
  • 1,107
  • 2
  • 11
  • 25
16
votes
1 answer

Is it possible to access estimator attributes in spark.ml pipelines?

I have a spark.ml pipeline in Spark 1.5.1 which consists of a series of transformers followed by a k-means estimator. I want to be able to access the KMeansModel.clusterCenters after fitting the pipeline, but can't figure out how. Is there a…
hilarious
  • 511
  • 3
  • 9
16
votes
6 answers

Is it possible to pipe conditionally in Powershell, i.e. execute an element of a pipeline only if a condition is met?

I want to do something like this: | | if | | | The results of run through , then they run through only if is met, then through the…
Adi Inbar
  • 12,097
  • 13
  • 56
  • 69
16
votes
2 answers

Wget file and send it to Bash

I want to make a Bash script which has to use Wget and run its output with Bash like this: wget -q -O - http://pastebin.com/raw.php?i=VURksJnn | bash The pastebin file is a test script, but this commands shows me: "Unknown command" (maybe due to…
Max13
  • 919
  • 2
  • 9
  • 27
15
votes
2 answers

How to set system path variable in github action workflow

I was wondering how I can set the system path variables in the GitHub actions workflow. export "$PATH:$ANYTHING/SOMETHING:$AA/BB/bin"
user15281199
15
votes
3 answers

How do I get my Azure DevOps Pipeline build to fail when my linting script returns an error?

I am using the Azure Pipelines GitHub add-on to ensure that pull requests pass my linting. However, I have just made a test pull request which fails my linting, but the Azure Pipeline succeeds. Here is my azure-pipelines.yml # Node.js with React #…
zagd
  • 361
  • 1
  • 2
  • 9
15
votes
2 answers

How many pipeline stages does the Intel Core i7 have?

How many instructions it can handle at a time ?
KFC
  • 541
  • 1
  • 4
  • 17
15
votes
1 answer

OpenCV Pipeline Editor

I am learning OpenCV (using python interface). I'm not really sure what I'm doing, so I keep adding and removing functions (blur, threshold, contours, edge detection) and modifying parameters. What would be very helpful is a UI that allows me to…
C Dorman
  • 551
  • 5
  • 12
15
votes
2 answers

AttributeError: lower not found; using a Pipeline with a CountVectorizer in scikit-learn

I have a corpus as such: X_train = [ ['this is an dummy example'] ['in reality this line is very long'] ... ['here is a last text in the training set'] ] and some labels: y_train = [1, 5, ... , 3] I would like to use…
tumultous_rooster
  • 12,150
  • 32
  • 92
  • 149