Questions tagged [pipeline]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

In a command line interface or shell, a pipeline uses the pipe operator ("|") to take output from one function or command and input it to another. This is done in a series like "command1 | function1 | command2". For questions related to the pipe operator use the tag.

In computer architecture, a pipeline is a process consisting of a sequence of stages that must be performed in serial order over each element passing the pipe, but may execute in parallel over the elements inside, such that the overall throughput does not depend on the length of the pipe. This is utilized by most CPUs' hardware to process instructions.

A similar technique is also done in software (software-pipelining) in order to optimize the parallelism of a given loop by reordering it to arrange data dependencies in a pipelined manner.

More broadly, "pipeline" is synonymous with "workflow."

See also:

5444 questions
35
votes
7 answers

"Piping" output from one function to another using Python infix syntax

I'm trying to replicate, roughly, the dplyr package from R using Python/Pandas (as a learning exercise). Something I'm stuck on is the "piping" functionality. In R/dplyr, this is done using the pipe-operator %>%, where x %>% f(y) is equivalent to…
Malthus
  • 568
  • 1
  • 7
  • 11
35
votes
7 answers

How do you determine if WPF is using Hardware or Software Rendering?

I'm benchmarking a WPF application on various platforms and I need an easy way to determine if WPF is using hardware or software rendering. I seem to recall a call to determine this, but can't lay my hands on it right now. Also, is there an easy,…
Charley Rathkopf
  • 4,720
  • 7
  • 38
  • 57
34
votes
4 answers

How to access scrapy settings from item Pipeline

How do I access the scrapy settings in settings.py from the item pipeline. The documentation mentions it can be accessed through the crawler in extensions, but I don't see how to access the crawler in the pipelines.
avaleske
  • 1,793
  • 5
  • 16
  • 26
33
votes
7 answers

Why am I getting "Pipeline failed due to the user not being verified" & "Detached merge request pipeline" on a Gitlab merge request?

When a non-owner dev pushes a branch to our Gitlab repo, it returns a "pipeline failed" message, with the detail "Pipeline failed due to the user not being verified". On the dev's account, he's getting a prompt to add a credit card to verify him to…
30
votes
2 answers

Extract the second element of a tuple in a pipeline

I want to be able to extract the Nth item of a tuple in a pipeline, without using with or otherwise breaking up the pipeline. Enum.at would work perfectly except for the fact that a tuple is not an enum. Here's a motivating example: colors = %{red:…
AnilRedshift
  • 7,937
  • 7
  • 35
  • 59
30
votes
4 answers

Run a program in a ForEach loop

I'm trying to get this simple PowerShell script working, but I think something is fundamentally wrong. ;-) ls | ForEach { "C:\Working\tools\custom-tool.exe" $_ } I basically want to get files in a directory, and pass them one by one as arguments to…
Luke Quinane
  • 16,447
  • 13
  • 69
  • 88
30
votes
4 answers

Do function pointers force an instruction pipeline to clear?

Modern CPUs have extensive pipelining, that is, they are loading necessary instructions and data long before they actually execute the instruction. Sometimes, the data loaded into the pipeline gets invalidated, and the pipeline must be cleared and…
abelenky
  • 63,815
  • 23
  • 109
  • 159
29
votes
6 answers

Assign intermediate output to temp variable as part of dplyr pipeline

Q: In an R dplyr pipeline, how can I assign some intermediate output to a temp variable for use further down the pipeline? My approach below works. But it assigns into the global frame, which is undesirable. There has to be a better way, right? I…
lowndrul
  • 3,715
  • 7
  • 36
  • 54
29
votes
11 answers

How to extract best parameters from a CrossValidatorModel

I want to find the parameters of ParamGridBuilder that make the best model in CrossValidator in Spark 1.4.x, In Pipeline Example in Spark documentation, they add different parameters (numFeatures, regParam) by using ParamGridBuilder in the Pipeline.…
Mohammad
  • 1,006
  • 2
  • 15
  • 29
29
votes
4 answers

how to use xargs with sed in search pattern

I need to use the output of a command as a search pattern in sed. I will make an example using echo, but assume that can be a more complicated command: echo "some pattern" | xargs sed -i 's/{}/replacement/g' file.txt That command doesn't work…
Neuquino
  • 11,580
  • 20
  • 62
  • 76
28
votes
1 answer

pipeline in docker exec from command line and from python api

What I try to implement is invoking mysqldump in container and dump the database into the container's own directory. At first I try command below: $ docker exec container-name mysqldump [options] database | xz > database.sql.xz That's not working,…
pingz
  • 555
  • 1
  • 10
  • 20
27
votes
10 answers

Need to perform AWS calls for account xxx, but no credentials have been configured

I'm trying to deploy my stack to aws using cdk deploy my-stack. When doing it in my terminal window it works perfectly, but when im doing it in my pipeline i get this error: Need to perform AWS calls for account xxx, but no credentials have been…
gospecomid12
  • 712
  • 3
  • 11
  • 25
27
votes
4 answers

-> operator in Clojure

Is the -> operator in Clojure (and what is this operator called in Clojure-speak?) equivalent to the pipeline operator |> in F#? If so, why does it need such a complex macro definition, when (|>) is just defined as let inline (|>) x f = f x Or if…
Dax Fohl
  • 10,654
  • 6
  • 46
  • 90
27
votes
7 answers

Singleton array array(, dtype=object) cannot be considered a valid collection

Not sure how to fix . Any help much appreciate. I saw thi Vectorization: Not a valid collection but not sure if i understood this train = df1.iloc[:,[4,6]] target =df1.iloc[:,[0]] def train(classifier, X, y): X_train, X_test, y_train, y_test =…
manisha
  • 455
  • 2
  • 7
  • 10
26
votes
3 answers

YAML_FILE_ERROR: YAML file does not exist

I'm trying to implement a pipeline on AWS, but I get an error: YAML_FILE_ERROR: YAML file does not exist I don't know why. I'm using github repo for mean stack project, entry file is docker-compose. Yml.
Abhishek
  • 1,742
  • 2
  • 14
  • 25