Questions tagged [pipeline]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

In a command line interface or shell, a pipeline uses the pipe operator ("|") to take output from one function or command and input it to another. This is done in a series like "command1 | function1 | command2". For questions related to the pipe operator use the tag.

In computer architecture, a pipeline is a process consisting of a sequence of stages that must be performed in serial order over each element passing the pipe, but may execute in parallel over the elements inside, such that the overall throughput does not depend on the length of the pipe. This is utilized by most CPUs' hardware to process instructions.

A similar technique is also done in software (software-pipelining) in order to optimize the parallelism of a given loop by reordering it to arrange data dependencies in a pipelined manner.

More broadly, "pipeline" is synonymous with "workflow."

See also:

5444 questions
1
vote
1 answer

cannot pass argument to which with pipeline in R

I want to filter columns of a data.table based on their attribute. The answer is actually based on Nera's answer here Convert column classes in data.table # example DT <- data.table(x = c("a", "b", "c"), y = c(1L, 2L, 3L), …
Ian Wang
  • 135
  • 8
1
vote
1 answer

Square brackets and spaces/special characters in table names in Copy Task in Azure Data Factory

Product: Azure Data Factory Component: Copy data Activity Issue Description: I am encountering an error when using Azure Data Factory's Copy data Activity to execute a SQL query. The issue arises when querying a table whose name contains spaces,…
Daniel A
  • 11
  • 1
1
vote
0 answers

How to force a GitLab pipeline job to fail when a HTTP request returns a status code other than 200?

I try to make a http request to some host and if http status code is not 200 GitLab pipeline job should fail. How I can make GitLab job to fail? But job succeeded. Here is script section of my gitlab-ci.yml: script: - 'if [ $(curl -X POST -s…
1
vote
1 answer

I need to aggregate two MongoBD collections

I need to aggregate from two tables into one, so that name from products and id_item, quantity from customer here is what node module do I use "dependencies": { "connect-mongo": "^5.0.0", "dotenv": "^16.0.3", "ejs": "^3.1.9", …
qruim
  • 33
  • 5
1
vote
0 answers

Troubleshooting: Azure pipeline PowerShell task throwing 502 Bad Gateway response

My pipeline for deploy webapp on Azure fails on inline task Powershell. Inside the script I make two web requests with: Invoke-RestMethod -Method POST -ContentType application/json The first one works and I use part of response data to make a…
1
vote
1 answer

Running docker inside a container in AWS CodeBuild

I got a Java application. In local, I can connect to Dev Container using Visual Studio Code. Now I wanna build a CodePipeline in AWS. But it displays an error like this when I tried to start docker in CodeBuild's Ubuntu standard 7.0…
julanove
  • 405
  • 5
  • 19
1
vote
1 answer

Having a problem building an ML.NET pipeline

I'm having a problem building an ML.Net pipeline. I've read through ALOT of Microsoft documentation, but I think the problem is I just don't understand it. Was wondering if I could get some help from this community? What I'm trying to do is to…
jason835
  • 19
  • 3
1
vote
1 answer

wildcard and dynamic filename on datafactory read file

ok I have to copy from a remote storage account a daily file. this file is generated with format name: File20230515063915.TXT meaning: "the word File"+year+month+date+hour, etc. everything on my time (+5) The thing is, the storage account have…
1
vote
1 answer

Sklearn pipeline with LDA and KNN

I try to use LinearDiscriminantAnalysis (LDA) class from sklearn as preprocessing part of my modeling to reduce the dimensionality of my data, and after applied a KNN classifier. I know that a good pratice is to use pipeline to bring together…
Adrien Riaux
  • 266
  • 9
1
vote
1 answer

Preprocessing data in TensorFlow

I have a simply sequential model written in Python using TensorFlow library. As an input I have categorical and numerical columns and in output I'm getting float number. I would like deploy my model in Windows Application (.NET) and I am wondering…
Mieczmik
  • 11
  • 1
1
vote
1 answer

Data factory: Inserting the body in a copy data activity source

I am trying to get data from a REST API. According to the documentation, the API needs to have a date in the body like this: {"date": "2023-01-05"} It's working when I try it in Postman. When I try it in Azure Data Factory, it doesn't work. I have…
Haroon
  • 15
  • 4
1
vote
1 answer

Manipulating with input/output jacks in Nextflow

I'm a newbie Nexflow user. And I'm struggling to familiarize input/output jacks in Nexflow. I knew that Nextflow has DAG visualisation, a useful feature for drawing a directed chart for flow. I have a silly small chart like this. I want to write a…
Rossy Clair
  • 175
  • 1
  • 6
1
vote
1 answer

how to create pipeline in databricks using delta live table to read data from kafka

read from kafka this worked raw_kafka_test = (spark.readStream .etc ) @dlt.table( table_properties={"pipelines.reset.allowed":"false"} ) def raw_kafka(): return raw_kafka_test read from delta live table not worked @dlt.table( …
1
vote
1 answer

Drop a step from a sklearn pipeline using the step name

How to remove a step from a sklearn pipeline using the step name? By position I know that it can be done: pipeline.steps.pop(n) But with a very large pipeline, it can be difficult to find the position of the step you want to remove.
Slevin_42
  • 87
  • 9
1
vote
1 answer

Decline a PR when pipeline gets failed

I am using bitbucket pipelines for PR raised pipeline, is there any way to write in bitbucket-pipelines.yml or in bitbucket settings to Cancel/Decline PR if pipeline gets failed at any step? There are variables like "BITBUCKET_PR_ID",…