Questions tagged [pipeline]

A pipeline is a sequence of functions (or the equivalent thereof), composed so that the output of one is input for the next, in order to create a compound transformation. Famously, a shell pipeline looks like "command | command2 | command3" (but use the tag "pipe" for this). It's also used in computer architecture to define a sequence of serial stages that execute in parallel over elements being fed into a pipe, in order to increase the overall throughput.

In a command line interface or shell, a pipeline uses the pipe operator ("|") to take output from one function or command and input it to another. This is done in a series like "command1 | function1 | command2". For questions related to the pipe operator use the tag.

In computer architecture, a pipeline is a process consisting of a sequence of stages that must be performed in serial order over each element passing the pipe, but may execute in parallel over the elements inside, such that the overall throughput does not depend on the length of the pipe. This is utilized by most CPUs' hardware to process instructions.

A similar technique is also done in software (software-pipelining) in order to optimize the parallelism of a given loop by reordering it to arrange data dependencies in a pipelined manner.

More broadly, "pipeline" is synonymous with "workflow."

See also:

5444 questions
1
vote
1 answer

Setting Crypto Library to UTF8 with NodeJS Pipe

I get a stream of data with a query over PostgreSQL, then I encrypt this data in JSON format with the help of cipher and send it directly to the user (chunk type). The problem here is that while cipher encodes, it does not encode the data as utf8. I…
Halil Han BADEM
  • 208
  • 1
  • 15
1
vote
0 answers

Converting an ArrayList to an array then passing it out of a pipeline function seems to add index values to the resultant array

I've read the PowerShell Team blog post Converting to Array and I've implemented the ToArray function it describes. This works. The article mentions performance could be improved by using an ArrayList instead of an array but doesn't include code…
Simon Elms
  • 17,832
  • 21
  • 87
  • 103
1
vote
0 answers

I was trying to use transformers by parsing the pipeline function for setiment analysis

While i was trying to perform sentiment analysis using transformers, then i parsed the pipeline function. Unfortunately, I ran into a runtime error: Failed to import transformers.models.distilbert.modeling_tf_distilbert because of the following…
1
vote
0 answers

Cypress OOM error when running in a Gitlab pipeline job

When I attempt run Cypress inside of a Gitlab CI/CD pipeline job, I always get an error at the start of my tests that says, "Failed to adjust OOM score of renderer with pid..." For the base image of the job, I'm using the latest image for…
user22303309
1
vote
1 answer

Sklearn BaggingClassifier doesn't work with a pipeline(preprocessor, KNeighborsClassifier)

Using sklearn, I have a pipleline that works perfectly and basically looks and works like that : model_1_KNeighborsClassifier = make_pipeline(preprocessor, KNeighborsClassifier()) model_1_KNeighborsClassifier.fit(X_train, y_train) But if I do…
1
vote
1 answer

MongoDB to Databricks Data Ingestion

I am working on creating a pipeline from MongoDB to Databricks. Based on my research there are two ways of doing it: MongoDB Change Streams MongoDB-Databricks Connecor for Structured Streaming. I am using Pyspark. I am doing this to get all the…
1
vote
1 answer

I want to copy files to AWS ec2 using buildspec.yml file, the 22 port is open for all the traffic. How to restrict the 22nd port for only codebuild?

I want to ssh to the aws ec2 server on port 22. The port is open for all traffic. I want to restrict the traffic for only aws codebuild pipeline. How I can do this? I want to ssh to the aws ec2 server on port 22. The port is open for all traffic. I…
1
vote
0 answers

Nodejs Stream hangs when it reaches await db query statement

Node.js pipeline with a ReadStream which reads data from a Postgres database using knex.js stream, then a Transform Stream which transforms the data, followed by another Transform Stream that batches the chunks with a record size of 500. Finally, a…
SB07
  • 11
  • 3
1
vote
1 answer

How can i access my react VITE ENV variables in production of an azure static web app?

locally i use a .env file and do the following: const value = import.meta.env.VITE_VALUETOGET console.log("value", value ) When i go to production however i use the azure portal configuration app settings and added VITE_VALUETOGET with the value,…
NadimCrypt
  • 53
  • 1
  • 8
1
vote
0 answers

Gitlab running in parallel based on the subfolder

I have previously implemented jobs that run in parallel when a change is done to a file. Validating the Data: stage: validation parallel: matrix: - TEAM: [ "data", "monitoring", "sales", …
Gabriel
  • 11
  • 2
1
vote
2 answers

Equivalent of Unix/Linux tail command for Powershell - that supports pipes?

I'm aware that Get-Content has a -Tail parameter, but as far as I can tell, there's simply no way to get it to accept piped input. So how do I get something like this to work? I just want to be able to select first X or last X lines from any…
Cardgage
  • 13
  • 2
1
vote
1 answer

Use input file in Jenkins Active Choices Reactive References plugin

I'm trying to implement a pipeline in which, depending on the choice of parameter, the user sees either text input or file input. I use Active Choices Parameter: First: First Parameter Second: First Parameter then i need to get this values in…
1
vote
1 answer

Can speculative execution of modern CPUs cross loop iterations?

Consider below loop (https://godbolt.org/z/z4Wz1aanK) that has no loop-carried dependence. Will modern CPU speculatively execute next iteration with previous one? if true, is loop expansion still necessary here? void bar(void) { for (int i = 0;…
Changbin Du
  • 501
  • 5
  • 11
1
vote
1 answer

Batch to call powershell command to read out all wifi networks including keys

I am trying to get all known WiFi SSIDs including the Password. When I use the following command called from a batch file, I can't get it to work. I get the following error: Can someone please help me? powershell -command "(netsh wlan show profiles)…
Markus
  • 41
  • 1
  • 4
1
vote
1 answer

PowerShell: RunSpace.SessionStateProxy.SetVariable is not setting variable

ConsoleApplication: class Program { static void Main() { using (var runSpace = RunspaceFactory.CreateRunspace()) { runSpace.Open(); runSpace.SessionStateProxy.SetVariable("Var1", "Alex"); …
Rookian
  • 19,841
  • 28
  • 110
  • 180