Questions tagged [pipeline]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

In a command line interface or shell, a pipeline uses the pipe operator ("|") to take output from one function or command and input it to another. This is done in a series like "command1 | function1 | command2". For questions related to the pipe operator use the tag.

In computer architecture, a pipeline is a process consisting of a sequence of stages that must be performed in serial order over each element passing the pipe, but may execute in parallel over the elements inside, such that the overall throughput does not depend on the length of the pipe. This is utilized by most CPUs' hardware to process instructions.

A similar technique is also done in software (software-pipelining) in order to optimize the parallelism of a given loop by reordering it to arrange data dependencies in a pipelined manner.

More broadly, "pipeline" is synonymous with "workflow."

See also:

5444 questions
1
vote
1 answer

How to skip a job instead of failing pipeline in Gitlab

I have built a pipeline where I only run the security-check job, if the image build was successful. But the build only runs on main or master branch. Therefore, when I push to branches which are not none of these two, the pipeline fails with the…
Ricardo Silva
  • 380
  • 4
  • 15
1
vote
1 answer

Databricks processed files

I am currently setting up a data pipeline in databricks. The situation is as follow: Incoming data comes as json-files. Data is being fetched asynchronously to the filestore. In case data is received multiple times a day, this is put into the same…
bluhub
  • 129
  • 1
  • 2
  • 10
1
vote
1 answer

Gitlab CI - is there a difference between a rule with when:always to a rule without?

I have a pipeline in Gitlab-CI and one of the jobs has the rule: rules: - if: "$CI_PIPELINE_SOURCE == "merge_request_event" Will there be a difference if I will change it to: - if: "$CI_PIPELINE_SOURCE == "merge_request_event" when:…
eyalruby
  • 15
  • 5
1
vote
0 answers

Implement Pipeline with context that has stages without context

I need to write ExecutePipeline(ctx context.Context, in In, stages ...Stage) Out, that need to pass all objects of the in channel through stages. The problem is that I don't understand how the pipeline can be stopped by the ctx context if Stage…
Demps
  • 21
  • 2
1
vote
2 answers

How run sklearn.preprocessing.OrdinalEncoder on several columns?

this code raise error: import pandas as pd from sklearn.compose import ColumnTransformer from sklearn.pipeline import Pipeline from sklearn.preprocessing import OrdinalEncoder # Define categorical columns and mapping dictionary categorical_cols =…
parvij
  • 1,381
  • 3
  • 15
  • 31
1
vote
1 answer

Detect last cmdlet in a powershell pipeline

I want to write a cmdlet that changes its behavior depending on its pipeline position. Simplyfied example: Get-XYZObject | Rename-XYZObject If the cmdlet is the last in the pipeline it should work directly on a database object. Get-XYZObject |…
Tobias Wollgam
  • 761
  • 2
  • 8
  • 25
1
vote
1 answer

What does the `maxReplicas` property mean in the pipeline yaml in Azure in context of the k8s deployment?

What does the maxReplicas property mean in the pipeline yaml in Azure in context of the k8s deployment? E.g. in this documentation the maxReplicas: 10 is mentioned. But there is no explanation about what it means. At least I was not able to find…
manymanymore
  • 2,251
  • 3
  • 26
  • 48
1
vote
1 answer

Data Fusion for xml-to-json transformation: "+ExitOnOutOfMemoryError" and "exited with a non-zero exit code 3. Error file: prelaunch.err"

When transforming an xml file to json, the Data Fusion pipeline, configured in Autoscaling mode up to 84 cores, stops indicating an error. Can anybody help me to make it work? The 100-pages Raw log file seems indicating that possible errors…
1
vote
1 answer

What object is a sklearn.pipeline.Pipeline that applies a ColumnTransformer actually fitting on when fit(X, Y) is called on it

I am trying to get an idea of the inner workings of a scikit learn Pipeline. Consider the below data set and pipeline construction. data = pd.DataFrame({ 'Name': ['Alice', 'Bob', 'Charlie'], 'Age' : [30, 40, 37], 'City': ['Amsterdam',…
gebruiker
  • 115
  • 5
1
vote
1 answer

Host key verification failed error for submodule in gitlab pipeline

I am working on CI/CD pipeline (GitLab) build and I want to update my submodule. ( In local git submodule update --init --recursive --remote works well). But the same thing in the pipeline gives the below error: Host key verification failed. fatal:…
1
vote
1 answer

field [o365audit] not present as part of path [o365audit.CreationTime]

I suddenly started to getting the error: field [o365audit] not present as part of path [o365audit.CreationTime] Integration: Office 365 Logs Version: 1.4.1 Agent Version: 7.17.3 I cannot get the logs, could you please help? I also tried to change…
terentius
  • 21
  • 4
1
vote
0 answers

AttributeError: 'bytes' object has no attribute 'predict': Using pickle to load a Machine Learning model

I have a trained Machine Learning model saved in GitHub repository with the name "trained_model.sav". I want to download it using Requests module of Python and use it in my Python file (with .py extension). When I run the following code: url =…
1
vote
1 answer

All, could you help me understanding the piping outcome in powershell

It must be obvious but I do not get it being fairly new in PowerShell, can anybody explain why I can not 'pipe' the outcome of the following commands further: Get-Process | Where-Object { $_.ProcessName -match 'acc' } | Select-Object -Property…
Vsn
  • 21
  • 3
1
vote
0 answers

Azure DevOps - release pipeline, task=Azure CLI invalid role-perms or scope mis-match

Having issues w/release pipeline (Azure CLI) that has me scratching my head... Scenario: Release pipeline / Azure DevOps / 1 task having a single inline AZ CLI script utilizing a service connection back to my Azure cloud instance. Service…
gto406
  • 589
  • 3
  • 9
1
vote
0 answers

error while seding email from jenkins job

''' //def email_message = "Sending email to .... ${email}" //echo email_message.toString() def email_response = mail (to: "${email}", subject: "Jenkins build:${result}: ${job}", body: "${result}: Job: ${job} \nMore Info can be…
Gaurav S
  • 121
  • 15
1 2 3
99
100