Stack Overflow
Stack Overflow

Questions tagged [gnu-parallel]

GNU parallel is a shell tool for executing jobs in parallel using one or more computers.

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel. more...

733 questions
4
votes
1 answer

Difference in slurm Job Array and Job Step performance

I am running a set of many parallel jobs in slurm (around 1000) and each of these has to be assigned to one CPU. Reading the slurm documentation I found this: Best Practices, Large Job Counts Consider putting related work into a single Slurm job…
Ignacio
  • 377
  • 3
  • 12
4
votes
3 answers

Getting syntax error using awk in parallel processing

I have 44 .tsv files in one folder and I want to calculate the number of intersect of each pairwise with intersect command of bedtools tool. each output file would have 4 columns and I just need to save only sum of value of column 4 in each output…
Nastaran Esfahani
  • 41
  • 3
4
votes
1 answer

GNU parallel arguments

From the example seq 1 100 | parallel -I @@ \ > 'mkdir top-@@;seq 1 100 | parallel -X mkdir top-@@/sub-{} How do -X , @@, {} work? Also, what will be the behavior when '1' or '.' is passed inside {}? Is /> used for redirection here? I was trying…
Dasha Sham
  • 73
  • 1
  • 8
4
votes
4 answers

how to do with GNU parallel what is an equivalent of 'read word1 word2'

I have a pipe that gives me lines of two quoted space separated strings. Using echo to give you an example of the pipe content: echo -e "\"filename1\" \"some text 1\"\n\"filename2\" \"some text 2\"" "filename1" "some text 1" "filename2" "some text…
Diego
  • 812
  • 7
  • 25
4
votes
1 answer

gnu parallel: combined use of --pipe and args

using --pipe -N I can send a given number of lines as an input of job started by parallel. But how can I accomplish to run several jobs with different arguments given with ::: on each chunk? Let's take this little input file: A B C D E …
finswimmer
  • 10,896
  • 3
  • 34
  • 44
4
votes
3 answers

Pass in array to GNU Parallel to replace for loop

 a) I want to run 2 scripts in parallel b) I want to my for loops within those scripts in parallel. Before I had this code: for year in 2000 2001 2002 2003; do echo $year" LST data being merged" cd $base_data_dir/$year # this is the part…
Tommy Lees
  • 1,293
  • 3
  • 14
  • 34
4
votes
1 answer

Using parallel and Imagemagick to convert images to gif

I have used a python script which generates plots using a data file. Now because of huge amount of data, I am getting images in the range of 5000 and using ImageMagick's convert on a sequential processing is taking a lot of time. I referred to this…
Vishwesh
  • 108
  • 2
  • 9
4
votes
2 answers

How to run 2 or more scripts from different directory in parallel

As of now I a using time parallel to run scripts in parallel. Example... First, I'll go to the directory where the scripts are located. cd $DIR Then, execute scripts time parallel ::: $script1 $script2 $script3 This works well. But what if the…
Jane S.
  • 215
  • 1
  • 4
  • 12
4
votes
1 answer

GNU parallel array argument

Suppose I have a python file test.py: import os class print_args(object): def__init__(self, x, y, z): self.x = x self.y = y self.z = z print(x) print(y) print(z) if __name__ == '__main__': …
ajohnrobertson
  • 267
  • 2
  • 10
4
votes
1 answer

Parallelize for loop in bash

I have the following snippet in my bash script #!/bin/bash for ((i=100; i>=70; i--)) do convert test.png -quality "$i" -sampling-factor 1x1 test_libjpeg_q"$i".jpg done How can i execute the for loop in parallel using all…
user2650277
  • 6,289
  • 17
  • 63
  • 132
4
votes
2 answers

Ubuntu terminal - using gnu parallel to read lines in all files in folder

I am Trying to count the lines in all the files in a very large folder under Ubuntu. The files are .gz files and I use zcat * | wc -l to count all the lines in all the files, and it's slow! I want to use multi core computing for this task and…
thebeancounter
  • 4,261
  • 8
  • 61
  • 109
4
votes
2 answers

GNU Parallel: Halt on success -or- failure

Is it possible to set a -halt condition (or multiple -halt conditions?) such that all jobs will be halted if any of them fail, regardless of the exit code? I want to monitor for an event (that I just triggered, separately, on a load balanced…
Dana Lacoste
  • 179
  • 1
  • 7
4
votes
1 answer

How to escape brace in gnu-parallel

I have a python script that I want to call using gnu-parallel this way: parallel run_script.py --outfile=/path/to/somewhere/{}/{}.nc --shift={} ::: 1 2 3 How can I escape the first curly brace in [--outfile] to be used for python string formatting…
Nicolas
  • 43
  • 3
4
votes
1 answer

Kill all jobs spawned by sem

#!/bin/bash for i in {1..3}; do sem --no-notice --id $$ -j+0 sleep 10 ";" echo Done done sem --no-notice --id $$ --wait Launch it, then how to make Ctrl+C to terminate all the sleeps?
Velkan
  • 7,067
  • 6
  • 43
  • 87
4
votes
1 answer

Multiple read from a txt file in bash (parallel processing )

Here is a simple bash script for HTTP status code while read url do urlstatus=$(curl -o /dev/null --silent --head --write-out '%{http_code}' "${url}" --max-time 5 ) echo "$url $urlstatus" >> urlstatus.txt done < $1 I am…
user7423959
  • 85
  • 9
1 2 3
…
48 49