Stack Overflow
Stack Overflow

Questions tagged [gnu-parallel]

GNU parallel is a shell tool for executing jobs in parallel using one or more computers.

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel. more...

733 questions
5
votes
2 answers

tracking status/progress in gnu parallel

I've implemented parallel in one of our major scripts to perform data migrations between servers. Presently, the output is presented all at once (-u) in pretty colors, with periodic echos of status from the function being executed depending on which…
Andrej
  • 180
  • 1
  • 2
  • 10
5
votes
2 answers

GNU parallel inheriting environment variable while executing a local script

Suppose I have foo.sh that calls bar.sh using parallel: # foo.sh #! /bin/bash parallel -N 3 bar.sh ::: $(seq 10) My bar.sh works like this: if there is an environment variable (e.g. DEBUG=1) set, then it will output lots of debug info. Ideally I…
Stephen Chu
  • 343
  • 2
  • 15
5
votes
3 answers

How to install GNU parallel (noarc.rpm) on CentOS 7

I want to install GNU parrallel on Centos 7 There is not much info to find. Can someone explain me how to do this? This is some useful info I found
user5558501
5
votes
2 answers

parallel check md5 file

I have a md5sum file containing lots of lines. I want to use GNU parallel to accelerate the md5sum checking process. In the md5sum, when no file input, it will take the md5 string from stdin. I tried this: cat checksums.md5 | parallel md5sum -c…
Ken
  • 3,922
  • 9
  • 39
  • 40
5
votes
4 answers

How to use rsync instead of scp in my below shell script to copy the files?

I am using scp to copy the files in parallel using GNU parallel with my below shell script and it is working fine. I am not sure how can I use rsync in place of scp in my below shell script. I am trying to see whether rsync will have better…
john
  • 11,311
  • 40
  • 131
  • 251
5
votes
1 answer

Calling GNU parallel from GNU parallel

What's the proper way to nest calls to GNU parallel? Silly example: seq 1 100 | parallel echo {} | parallel seq {} 1000 My understanding is that on an 8-CPU box, each parallel would launch 8 jobs for a total of 64 jobs. If you're calling something…
Thomas Johnson
  • 10,776
  • 18
  • 60
  • 98
5
votes
1 answer

BASH...GNU Parallel

Racking my brains and scraping Google to find a way to get the sem command with timeout option to give me an exit code on timeout. It doesn't have to be through sem, it's just a pain because I need to end up applying actions differently based on…
264nm
  • 725
  • 4
  • 13
5
votes
1 answer

Executing GNU Parallel within a script

The command parallel echo {} ::: A B C executed from the command line return the correct result, while when invoked within a bash script return the error: This is the script: #script.bash #!/usr/bin/env bash parallel echo {} ::: A B C This is the…
memecs
  • 7,196
  • 7
  • 34
  • 49
5
votes
1 answer

GNU parallel: how to format replacement string?

I am trying to achieve the following and I want to do it on multiple processes using GNU parallel. for i in $(seq 0 3); do var=$(printf "%.5d" $i) echo test_$var done Output: -------------------- test_00000 test_00001 test_00002 I tried this…
memecs
  • 7,196
  • 7
  • 34
  • 49
5
votes
1 answer

GNU parallel show jobs remaining

I guess this question/problem can be tackled by the satisfaction of one (or more) of the below problems 1) How do I show jobs remaining? 2) How do I pretty the output of --eta 1) I've checked the man page, and I am presently using $PARALLEL_SEQ in…
Felipe Alvarez
  • 3,720
  • 2
  • 33
  • 42
4
votes
1 answer

Parallelize a awk script with multiple input files and changing the name of the output file

I have a series of text files in a folder sub.yr_by_yr which I pass to a for loop to subset a Beagle file from the header. I want to parallelize this script to subset the Beagle file from the header values (which is done using my subbeagle.awk…
M. Beausoleil
  • 3,141
  • 6
  • 29
  • 61
4
votes
1 answer

how to run bash for loop and using GNU parallel?

I have a bash loop where I am passing variables to a script. I want to run these in parallel with GNU parallel for FILE_NAME in FILE1 FILE2 FILE3; do ./SCRIPT -n $FILE_NAME done where I want the scripts to run in parallel as follows: …
Maria
  • 73
  • 6
4
votes
4 answers

How can I use the parallel command to exploit multi-core parallelism on my MacBook?

I often use the find command on Linux and macOS. I just discovered the command parallel, and I would like to combine it with find command if possible because find command takes a long time when we search a specific file into large directories. I…
user1773603
4
votes
1 answer

Optimal number of threads for GNU parallel

I think I have a fairly basic question. I just discovered the GNU parallel package and I think my workflow can really benefit from it! I am using a loop which loops through my read files and generates the desired output. The command that is…
nhaus
  • 786
  • 3
  • 13
4
votes
2 answers

How to run an Inotify shell script as an asynchronous process

I have an inotify shell script which monitors a directory, and executes certain commands if a new file comes in. I need to make this inotify script into a parallelized process, so the execution of the script doesn't wait for the process to complete…
Beeti Sushruth
  • 321
  • 2
  • 12
1 2 3
…
48 49