Stack Overflow
Stack Overflow

Questions tagged [gnu-parallel]

GNU parallel is a shell tool for executing jobs in parallel using one or more computers.

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel. more...

733 questions
4
votes
2 answers

bash loop in parallel

I am trying to run this script in parallel, for i<=4 in each set. The runspr.py is itself parallel, and thats fine. What I am trying to do is running only 4 i loop in any instance. In my present code, it will run everything. #!bin/bash for i in…
BaRud
  • 3,055
  • 7
  • 41
  • 89
4
votes
1 answer

How can I include the private key when running AWS jobs with GNU parallel?

How can one run parallel jobs on Amazon AWS EC2 with gnu parallel? I need to include private key to log in namely something like this does not work: sh script.sh | parallel --sshlogin a@b.com -i "key.pem"
Jane Maths
  • 43
  • 2
4
votes
2 answers

How To Avoid SIGCHLD error In Bash Script That Uses GNU Parallel

I'm running a script.sh in a loop. The script contains a parallel wget command. I'm getting the following error: Signal SIGCHLD received, but no signal handler set. The loop looks like this: for i in {1..5}; do /script.sh; done And the line that…
DomainsFeatured
  • 1,426
  • 1
  • 21
  • 39
4
votes
2 answers

Bash wait does not work

Part of my code: for i in "${r_arr[@]}" do ${parallel_dir}parallel -j${cap} --semaphore --semaphorename a2_bootstrap_semaphore_${rnd} a2_exec ${i} ${temp_zmat_folder_path}${temp_zmat_folder}/ ${outs_folder} done wait elapsed_time=$(($SECONDS -…
alex
  • 10,900
  • 15
  • 70
  • 100
4
votes
1 answer

GNU Parallel - Detecting that a command run in parallel has completed

So I have a situation where I'm running numerous commands with parallel and piping the output to another script that consumes the output. The problem I'm having is that my script that does the processing of output needs to know when a particular…
Brandon Wigfield
  • 137
  • 8
4
votes
1 answer

gnu parallel: space interpretation and within process variable editing

I'm having two problems with gnu parallel. Firstly the most interesting: I have a file in which one line contains two arguments separated by a space. These arguments should be passed to the command together, in a way that the command can recognize…
Laura
  • 105
  • 5
4
votes
1 answer

GNU Parallel: How do determine job "slot" you're using?

I'm trying to find a way to determine the job "slot" or "core" a command is currently using in parallel. For example, we've all seen a similar image of how parallel distributes commands: If I want to know which column a certain process is in, how…
Kevin Cupp
  • 497
  • 4
  • 10
4
votes
1 answer

How to assign variables in gnu parallel?

I want to assign a variable in the command that I pass to parallel: parallel "my_variable={}_33; echo $my_variable" ::: a b c The output should be: # a_33 # b_33 # c_33 Of course, this is just a toy example. In the real example I want to do other…
nachocab
  • 13,328
  • 21
  • 91
  • 149
4
votes
3 answers

Accessing Associative Arrays in GNU Parallel

Assume the following in Bash: declare -A ar='([one]="1" [two]="2" )' declare -a ari='([0]="one" [1]="two")' for i in ${!ari[@]}; do echo $i ${ari[i]} ${ar[${ari[i]}]} done 0 one 1 1 two 2 Can the same be done with GNU Parallel, making sure to…
Larry
  • 310
  • 2
  • 11
4
votes
1 answer

How to use GNU parallel on Multiple Computers

I wanted use GNU parallel on my two computers. I was successful at running parallel on one computer, but I was unable to run parallel on remote computer. Version: Local: parallel-20140222 , Remote: parallel-20130522. I had enabled password less ssh…
aravind ramesh
  • 307
  • 2
  • 15
4
votes
2 answers

GNU parallel used with xargs and awk

I have two large tab separated files A.tsv and B.tsv, they look like (the header is not in the file): A.tsv: ID AGE User1 18 ... B.tsv: ID INCOME User4 49000 ... I want to select list of IDs in A such that 10=< AGE <=20 and select…
Bamqf
  • 3,382
  • 8
  • 33
  • 47
4
votes
1 answer

GNU parallel does not run in parallel on remote servers when using --onall

I'm trying to run a script on multiple remote servers, with multiple parameters. The GNU parallel command is: parallel --onall -S ${RH32},{RH64} /shared/loc/script.sh ::: param1 param2 script.sh: host=`uname…
Omer Dagan
  • 14,868
  • 16
  • 44
  • 60
4
votes
2 answers

File I/O in gnu parallel

I have a program that takes a single argument. I am using gnu parallel to perform parameter sweeps on this argument. Each run generates a single result, and I want to append all results into a single file, say Results.txt. What would be a correct…
Neha Karanjkar
  • 3,390
  • 2
  • 29
  • 48
4
votes
1 answer

ffmpeg and gnu parallel

My work would require me to encode a few thousand movies in a few days. Each movie needs to be encoded in 3 different formats. I use ffmpeg to output these formats in parallel with a single read of the input source as detailed here:…
souvik
  • 51
  • 3
4
votes
1 answer

GNU Parallel: Delay first jobs sent to remote servers

I have a gnu parallel script that imports data (100,000 jobs distributed to 100 remote servers) into a central database. The first jobs are slamming the central db because they finish almost at the same time. The jobs after that eventually spread…
Nick
  • 555
  • 5
  • 22
1 2 3
…
48 49