Stack Overflow
Stack Overflow

Questions tagged [gnu-parallel]

GNU parallel is a shell tool for executing jobs in parallel using one or more computers.

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel. more...

733 questions
0
votes
1 answer

Memory leak with GNU Parallel extensions

I have a memory leak that only appears when I compile with OpenMP support and GNU Parallel extensions: -fopenmp -DGLIBCXX_PARALLEL. Example : std::transform(buffer, buffer + _tabLength, _origData, MathUtil::ShortToDouble); Result (sorry for the…
Jean-Michaël Celerier
  • 7,412
  • 3
  • 54
  • 75
0
votes
1 answer

How to launch multiple child processes that will automatically receive signals sent to parent

I want to create a Bash script to launch parallel child processes. Is there a way to do this so that the child scripts still receive signals that are sent to the parent process? Here's roughly how I want to launch child processes, but this doesn't…
Brent Bradburn
  • 51,587
  • 17
  • 154
  • 173
-1
votes
1 answer

Using Parallel with a sed command iterating over thousands of files

I have 100,000s of files that I wish to iterate the below sed command over: sed -s -i -e 's/[[:space:]].*//' -e '1 s/^/>/g' -e '3 s/|*//g' -e '3 s/^/>ref/g' -e '1h;2H;1,2d;4G' So far, I have been using a bash loop: for i in read_* ; do sed -s…
SaltedPork
  • 345
  • 3
  • 16
-1
votes
1 answer

Is it possible to use GNU parallel with a yes/no prompt for each job?

I have many files that I would like to concatenate. Here is an example which generates those file names with some example content: for lane in $(seq 1 4) do for sub in $(seq 1 8) do echo "sublib ${sub}, lane ${lane}, R1" >…
Danny
  • 101
  • 1
  • 8
-1
votes
1 answer

How to run compression in gnu parallel?

Hi I am trying to compress a file with the bgzip command bgzip -c 001DD.txt > 001DD.txt.gz I want to run this command in parallel. I tried: parallel ::: bgzip -c 001DD.txt > 001DD.txt.gz but it gives me this error: parallel: Error: Cannot open…
user2110417
-1
votes
1 answer

GNU parallel log from remote hosts

I'm trying to run clamsccan using parallel. path /mnt/data in mounted on each host from clamscan_servers find /mnt/data -type f | parallel --sshloginfile clamscan_servers --jobs 32 --sshdelay 0.1 --results parallel_outdir clamscan --max-scansize=0…
psmith
  • 1,769
  • 5
  • 35
  • 60
-1
votes
2 answers

Using GNU parallel command with gfind to gain in runtime for gupdatedb tool

I make follow to the previous post combine parallel and gfind I would like to build the gupdatedb database, containing all from main root / excepted the PRUNEPATHSlisted more below. I am working on MacOS 10.15 Catalina. So, I tried to modify the…
user1773603
-1
votes
1 answer

How does the scratch space differ from the normal disk space in the home node disk space?

I am new to HPC and I am struggling in setting up scratch space. In the cluster I am working with, I need to set-up Scratch space using the SLURM workload manager. And I am struggling with the following questions? How does the scratch space differ…
mathquest
  • 31
  • 3
-1
votes
1 answer

mpich pass argument per CPU used

Given: 2 Ubuntu 16.04 machines with multiple CPU cores. I want to execute multiple instances of program fixed_arg arg2on the machines, passing one file name per call as arg2 to the program. So far, working with xargs, this works on a single…
user7677771
  • 69
  • 8
-1
votes
1 answer

Can gnu parallel be used with the cat command and large arrays?

I am working on processing a long list of data from a file using bash. There are over 300,000 lines in this file, so using gnu parallel could cut down processing time significantly. In addition to the main data file, I am using a second, smaller…
gwilymh
  • 415
  • 1
  • 7
  • 20
-1
votes
1 answer

Parallelizing a for Loop which accesses files

Here is the complete code. In BER_SB, values of K,SB passed to rand-src command and value of sigama passed to transmit command are being calculated in main. Vlues written to BER array by BER_SB are being further used in main. BER_SB() { …
user4782995
  • 11
  • 1
-2
votes
1 answer

Copy multiple files in parallel

I have a directory which has some 10000 files and I wanted to copy those files in parallel. Since, I have 4 processors, I wanted to see if I could copy four files at the same time in parallel. I don't know how I can achieve this with GNU Parallel.…
Shawn Brar
  • 1,346
  • 3
  • 17
-2
votes
2 answers

filter lines based on 4th column value having difference greater than 80 from previous and next line

I have a file like this having about 20 million rows. 11.tsv SDSS01001000.1 M1 100021 11.tsv SDSS01001000.1 M1 100082 11.tsv SDSS01001000.1 M1 100140 11.tsv SDSS01001000.1 M1 100270 11.tsv SDSS01001000.1 M1 …
onkar
  • 29
  • 7
1 2 3
…
48
49