Stack Overflow
Stack Overflow

Questions tagged [gnu-parallel]

GNU parallel is a shell tool for executing jobs in parallel using one or more computers.

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel. more...

733 questions
0
votes
0 answers

Limit of terminal processes?

I have been running the same script in different terminal windows using different arguments. I have accumulated about 40 terminal windows. I am aware of using '&' to run all of these scripts in one window, and also GNU parallel. Though I am open to…
Labrat
  • 105
  • 10
0
votes
1 answer

shell script to excute c++ program in parallel to different machine without asking password

I need to write shell script to execute c++ program in centos linux multiple machine parallel without asking password,so can I write please help me thanks for your response. I am try to write like this but this is not parallel and asking…
Sisay Zinabu
  • 71
  • 2
  • 14
0
votes
2 answers

Renaming .jpg files in the sub-folders

I have a large number of files in sub-folders that I need to rename. For example, I have: ParentFolder/sub-Folders/*.jpg How can I copy the files with a new naming convention as follows? ParentFolder1.jpg ParentFolder2.jpg
phh
  • 149
  • 1
  • 2
  • 10
0
votes
0 answers

optimize parallelization of line-by-line bash function

I have dozens of files that are 2-10GB in size that need re-formatting, and the bash script I have currently takes >1 day for a single file. Redirecting the file to 3< seems like more overhead, so I commented that approach out. I have several other…
BlacquenedRed
  • 73
  • 7
0
votes
1 answer

Jobs run by GNU parallel don't seem to be parallel

I wanted to use GNU parallel tool to make rsync file transfer to several hosts faster, but it seems that the parallel tool doesn't make it parallel much. I expected time parallel "sleep 5; echo {}" ::: 1 2 3 to take not much longer than 5 seconds,…
mitroo
  • 604
  • 5
  • 8
0
votes
2 answers

GNU parallel to keep docker-compose in attached mode

I'm trying to run a bunch of docker-compose commands in parallel using GNU parallel. For some reason though, it looks like starting parallel force docker-compose in detached mode so I can't access the container output anymore. I was hoping parallel…
Simon Boudrias
  • 42,953
  • 16
  • 99
  • 134
0
votes
1 answer

How to wget files and save as domain name?

Here's what I'm currently running to wget files: parallel -a list.txt --jobs 100 wget -P /home/files/ The list.txt file contains a list of files such as this: example.com/test.html anotherexample.com/test.html sample.com/test.html However, it…
Jamie
  • 91
  • 2
  • 5
0
votes
1 answer

persist variable between gnu parallel processes

I have a simple script like this: #!/usr/bin/env zsh count=0 while read -A words do # set count for processed records ((count++)) printf ":%d:\n" $count .. .. .. # go find something in somefile grep "$regex"…
branquito
  • 3,864
  • 5
  • 35
  • 60
0
votes
1 answer

How to distribute executions over a cluster

I am doing research, and I often need to execute the same program with different inputs (each combination of inputs repeatedly) and store the results, for aggregation. I would like to speed up the process by executing these experiments in parallel,…
marcotama
  • 1,991
  • 2
  • 19
  • 24
0
votes
0 answers

GNU Parallel --sshlogin remote shells

I have a small test setup of 5-10 embedded devices, and would like to remotely run commands on them with GNU Parallel. But the commands I supply produce no output. Copying and pasting the command output by parallel -vv ... into the zsh login shell…
Will Angley
  • 1,392
  • 7
  • 11
0
votes
1 answer

How to set variables in a subshell?

I have a script which will be running in each server and copies certain files into it. Script knows where I am running and what files I need to copy. Script will copy files from local datacenter local_dc but if it is down or not responding, then it…
john
  • 11,311
  • 40
  • 131
  • 251
0
votes
0 answers

Why is object file empty after archiving when building in parallel with GNU Make?

I have a problem with the archiver not seeming to archive the object files fully correctly every time when building in parallel. I have built the project with make -j32 If I investigate the archive: ar x mylib.a ls -l myFile.o -rw-rw-r-- 1 tester…
Euklides
  • 564
  • 1
  • 10
  • 35
0
votes
1 answer

Can GNU make create broken binaries when building in parallel?

I am working in a project where we have just added parallelism to our build system, using GNU Make. We build both libraries and the programs in parallel. First we build all the libs necessary for the binaries. After the libs are created we start…
Euklides
  • 564
  • 1
  • 10
  • 35
0
votes
2 answers

Speeding up zgrep to pull matched lines out of compressed json files

I've got about 100k gzipped JSON files that together equal about 100GB. All the files are in the same directory. I'm running this locally on my mac os x. I have several different patterns to match, and have tried running the command a variety of…
tchaymore
  • 3,728
  • 13
  • 55
  • 86
0
votes
2 answers

Problems with Perl installation wrong ELF class: ELFCLASS32

I wanted to run gnu_parallel on one of my servers(nproc=48). When I try to run the command, seq 1 5000 | parallel --eta 'echo {} > /dev/null' I was getting some errors, Can't load '/usr/local/lib64/perl5/auto/Fcntl/Fcntl.so' for module Fcntl:…
aravind ramesh
  • 307
  • 2
  • 15
1 2 3
…
48
49