Questions tagged [pipeline]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

In a command line interface or shell, a pipeline uses the pipe operator ("|") to take output from one function or command and input it to another. This is done in a series like "command1 | function1 | command2". For questions related to the pipe operator use the tag.

In computer architecture, a pipeline is a process consisting of a sequence of stages that must be performed in serial order over each element passing the pipe, but may execute in parallel over the elements inside, such that the overall throughput does not depend on the length of the pipe. This is utilized by most CPUs' hardware to process instructions.

A similar technique is also done in software (software-pipelining) in order to optimize the parallelism of a given loop by reordering it to arrange data dependencies in a pipelined manner.

More broadly, "pipeline" is synonymous with "workflow."

See also:

5444 questions
1
vote
1 answer

Custom Transformers in Sklearn Pipeline do not work as expected

I am working on ML project using sklearn. I have writtern few custom transformers as below: DateTimeTransformer - To extract day, month, year, hour, minute, second (thereby getting 6 new columns) applied on Arrival Time KBinTransformer - To turn…
winter
  • 467
  • 2
  • 10
1
vote
0 answers

Unable to Infer Spark ML Pipeline model when built using Custom Preprocessing Stages

Unable to infer a Spark ML Pipeline model built using Custom Transformers/Estimators. I had some custom requirements to transform a raw data. Those custom operations were not in the pyspark.ml module. In order to facilitate these operations, I…
1
vote
1 answer

Stream powershell output objects to format-table

I'm writing a powershell function to process lines from a log file. The log file is being written to by another process. In order to make viewing the logs easier I'm parsing each log line into an object and passing that along the pipeline. I've…
Ben Randall
  • 1,205
  • 10
  • 27
1
vote
1 answer

iterating through a file in a Nextflow process

I am working with nextflow to create a pipeline, and I am facing some problems in one of the processes. I have a process that takes as input 2 normal files (output.kraken, and $sequences) and a string ("Aspergillus" for example) I have another file…
1
vote
1 answer

Is it possible to run a Gitlab CI pipeline using multiple gitlab releases or branches

I want to be able to have a pipeline that deploys a main branch to a test environment and then a versioned release to production. The versioned prod env will only be have a newer versioned deployed once testing has been signed off. I want to have a…
1
vote
0 answers

Is there a Julia equivalent of the targets R package?

The targets R package lets the user define an execution pipeline with minimal extra syntax and automatically handles things like cache and random seeds. It would be nice to have a similar tool in Julia. Closest thing I have found so far is…
1
vote
0 answers

Issue with decoding H264 frames using appsrc in GStreamer

I am currently facing an issue while attempting to decode H264 frames using the appsrc element in GStreamer. I have constructed the following pipeline within my application: appsrc name=mysource ! video/x-h264,…
isra60
  • 512
  • 1
  • 6
  • 18
1
vote
1 answer

Sklearn Pipeline - Customized 'Optional Estimator'

I have created this function below, that creates a pipeline and returns it. def make_final_pipeline(columns_transformer, onehotencoder, estimator, Name_of_estimator, index_of_categorical_features, use_smote=True): if use_smote: # Final…
1
vote
0 answers

Swift Package Manager does not support provisioning profiles

I had a pipeline configured in azure devops that uploaded apps to testflight, but when I updated to xcode 14, my pipeline has returned the following error for SPMs with resources: note: Building targets in dependency…
Yris
  • 11
  • 1
1
vote
1 answer

Azure Load testing cannot create a new test

I have a yaml file for test config as follows: version: v0.1 testId: NAME testPlan: sampleTest description: 'Load Test' engineInstances: 1 failureCriteria: - avg(response_time_ms) > 15000 - percentage(error) > 10 and in my pipeline yaml file I…
ht1994G
  • 11
  • 2
1
vote
0 answers

How to treat warnings as errors in Azure DevOps Pipeline VSBuild task?

A request came in to treat all warnings as errors in our build pipelines. The expectation is that it will report all as errors and will stop the pipeline from getting the green check mark. After a bit of research, I thought I could just apply a…
1
vote
1 answer

Is there a simpler way to do this pipeline with languageext?

I'm trying to grok how to deal with functions that turn one "Either" into many "Either"s and how you then merge those back into a single stream. The following turns one string into many numbers, squaring each and ignoring any errors. Given…
1
vote
0 answers

How to pass parameter to asset from op in dagster

I am new to dagster and having difficult in running the asset for different parameter values when job scheduled. I have created a pipeline using dagster. Trying to materialize the outcome of upstream asset multiple_num() and using op to pass…
Hari
  • 299
  • 4
  • 12
1
vote
0 answers

Tests failing in bitbucket's pipeline

I am working with tests involving date for the first time, and needed to test if a function converted a string to the date format with the expected timezone (America/Sao Paulo) for me to persist in my database, however these tests are successfully…
1
vote
0 answers

Determine Individual Jenkins stage result in-case of Parallel execution of Stages

I am stuck in peculiar problem. I have a complex Jenkins pipeline with multiple stages running in parallel. Each stage is wrapped in a catchError block, so that all stages run irrespective of results. I want to capture the results of individual…