Stack Overflow
Stack Overflow

Questions tagged [gnu-parallel]

GNU parallel is a shell tool for executing jobs in parallel using one or more computers.

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel. more...

733 questions
3
votes
1 answer

GNU parallel not spawning jobs

After an upgrade to Debian 8.6 Jessie the GNU parallel script suddenly stopped parallelizing to more than 2 jobs with the --pipe and -L options. Before the upgrade the command: cat file_with_1064_lines.txt | parallel -L10 -j5 -k -v --pipe "wc…
artem
  • 95
  • 5
3
votes
2 answers

How to use the arguments from `xargs` as the index of an array

I want to use the aruments from the xargs as the index of this array, this is the scripts: 1 #!/bin/bash 2 array[0]=x 3 array[1]=y 4 echo array : ${array[0]}, ${array[1]} 5 echo -n {0..1} | xargs -I index -d" " echo index,${array[index]} and this…
spring cc
  • 937
  • 1
  • 10
  • 19
3
votes
2 answers

GNU Parallel | pipe command

I am completely new in using GNU parallel and I need your advice in running the command below using GNU parallel: /home/admin/Gfinal/decoder/decdr.pl --gh --w14b /data/tmp/KRX12/a.bin | perl /home/admin/decout/decoder/flow.pl >>…
Helmy
  • 115
  • 1
  • 3
  • 10
3
votes
1 answer

how to set number of jobs for GNU parallel in my script

In the following script I want to explicitly set number of concurrent jobs to run the command: #!/usr/bin/parallel -j 2 --shebang-wrap /bin/bin echo hi "$@" when I run the script, I get the following error: parallel: Error parsing of…
ayyoob imani
  • 639
  • 7
  • 16
3
votes
1 answer

How to tar files with a size limit?

I am working on taking backup my server data. Some folders have data around 600GB, I need to tar it as 6 files for 100GB each. I have google it got some idea to do it.(similar topic#1, similar topic#2 and so). we can achive it by tar cvzf - data/ |…
saravanakumar
  • 1,747
  • 4
  • 20
  • 38
3
votes
2 answers

bash loop with GNU less that refreshes every 2 minutes

I have a linux machine with a data folder and a script that produces a report xlsx spreadsheet file on the ever changing content of the folder where it is being run. I use a xlsx2tsv script to convert it to tsv text which takes less than one second.…
719016
  • 9,922
  • 20
  • 85
  • 158
3
votes
3 answers

How to use gnu-parallel for processing a script with two inputs?

I am trying to run a Python script with two inputs as follows. I got ~300 of these two inputs so I wonder if somebody could advise how to run them with parallel. The single run looks like: python stable.py KOG_1.fan KOG_1.fasta > KOG_1.stable My…
Liliana Cano
  • 41
  • 3
3
votes
2 answers

Grepping a 1M row file with 320K patterns stored in another file

I tried to grep a 1M row '|' separated file with 320K patterns from another file with piping to Ole Tange's parallel package and piping the matched results into another file. I am using Cygwin on Windows 7 with 24 cores and 16GB physical memory. The…
andrnev
  • 410
  • 2
  • 12
3
votes
2 answers

How to use shell variables with GNU parallel?

This is the content of list.csv: Apple,Red,10 Banana,Yellow,3 Coconut,White,18 Suppose I have this GNU parallel command: parallel -a list.csv -j0 -C, \ color=`echo {2} | sed 's/e/eee/g' | ./capitalize.sh` ";" \ echo "{2}" ";" \ echo "$color"…
octosquidopus
  • 3,517
  • 8
  • 35
  • 53
3
votes
1 answer

How best to use GNU parallel for jobs using a lot of CPU time

Im trying to write an more efficient pipeline for my ChIP-seq analysis - after I discovered GNU parallel (thumbs up). my script is long and enters a bottleneck when I have to call peaks with a program called MACS2 ( https://github.com/taoliu/MACS/…
I. mall
  • 31
  • 1
3
votes
1 answer

'parallel' or 'find' to iterate over directories? [bash]

In order to save space on my back-up disk, I want to "mothball" the data files that can be easily regenerated and thus don't need to be backed-up. Currently, I'm using UNIX's "parallel" command to essentially split a large nested for-loop over many…
yunque
  • 625
  • 1
  • 8
  • 18
3
votes
1 answer

How do I create a Stack or LIFO for GNU Parallel in Bash

While my original problem was solved in a different manner (see comment thread under this question, as well as the edits to this question), I was able to create a stack/LIFO for GNU Parallel in Bash. So I will edited my background/question to…
Jake
  • 625
  • 6
  • 16
3
votes
5 answers

Installing GNU parallel without root permission

EDIT 2 After trying to run parallel, I am getting the following on the terminal when I run both parallel and by the source /homedtic/gsantamaria/parallel/bin/parallel: perl: warning: Setting locale failed. perl: warning: Please check that your…
HackCode
  • 1,837
  • 6
  • 35
  • 66
3
votes
2 answers

Change $TMPDIR in a parallel processing

I run a batch job on a high-performance computing system to sort aligned reads and used GNU parallel to speed up my work, but my job failed with the following reason: parallel: Error: Output is incomplete. Cannot append to buffer file in $TMPDIR.…
lisa
  • 69
  • 1
  • 4
3
votes
1 answer

gnu parallel missing argument to bash?

When I try to use gnu parallel's semaphore, any command results in: /bin/bash: -c: option requires an argument For instance: parallel --semaphore --verbose sleep 1000 /bin/bash -c /bin/bash: -c: option requires an argument Why isn't anything…
Kyle
  • 224
  • 2
  • 11
1 2 3
…
48 49