Stack Overflow
Stack Overflow

Questions tagged [gnu-parallel]

GNU parallel is a shell tool for executing jobs in parallel using one or more computers.

GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel. more...

733 questions
4
votes
2 answers

Executing SCP in GNU Parallel

I need to run some commands in parallel including SCP. For this I'm using GNU Parallel. The problem is I don't know how to pass the password to SCP. This is a line similar to the one I'm running: ls 2011_* | parallel scp {} user@domain And if ls…
Topo
  • 4,783
  • 9
  • 48
  • 70
4
votes
2 answers

GNU parallel load balancing

I am trying to find a way to execute CPU intensive parallel jobs over a cluster. My objective is to schedule one job per core, so that every job hopefully gets 100% CPU utilization once scheduled. This is what a have come up with so far: FILE…
user000001
  • 32,226
  • 12
  • 81
  • 108
3
votes
1 answer

How do you get GNU Parallel to parse quoted command line arguments?

This is one sample program in the GNU parallel documentation for executing via the shell script shebang. #!/usr/bin/parallel --shebang-wrap --colsep " " /bin/bash echo Arguments: $@ The output for ./bash_echo.parl gracias 'buenos…
Linus Fernandes
  • 498
  • 5
  • 30
3
votes
1 answer

Cuda AtomicAdd for int3

In Cuda AtomicAdd for double can be implemented using a while loop and AtomicCAS operation. But how could I implement an atomic add for type int3 efficiently?
Nyaruko
  • 4,329
  • 9
  • 54
  • 105
3
votes
1 answer

GNU Parallel and Python atexit

I was trying to run a python script with GNU parallel. Everything seems to work, except for the atexit routine used inside the python script. It seems, after ctrl+c, parallel is killing the python process without giving python a chance to call the…
Axeon Thra
  • 334
  • 3
  • 14
3
votes
2 answers

How to redirect sequence of numbers to parallel in bash

I wanted to parallelize curl requests and i used the code here. The input i want to use is a sequence of numbers generated using seq but the redirection keeps giving me errors like ambiguous input. Here is the code: #! /bin/bash brx() { num="$1" …
kenjoe41
  • 280
  • 2
  • 8
3
votes
1 answer

GNU parallel loses pv progress output

It seems if I use GNU Parallel to wrap a command that uses the pv (pipe viewer) command, it loses the command's progress output. For example, if I run the following pv command, which shows the progress of my gzip: › dd if=/dev/urandom bs=256m…
Stephen Chu
  • 343
  • 2
  • 15
3
votes
3 answers

Split file into several files based on condition and also number of lines approximately

I have a large file with a sample as below A222, 00000, 555 A222, 00100, 555 A222, 00200, 555 A222, 00300, 555 A222, 00000, 555 A222, 00100, 555 A222, 00000, 555 A222, 00200, 555 It's a sample file which has order header(00000) and related order…
Arpit Singh
  • 95
  • 1
  • 3
3
votes
1 answer

How to print stdout & stderr of failed job in GNU parallel?

I'm invoking GNU parallel on a bunch of commands using parallel --keep-order --line-buffer --halt 2. Every once in a while one of the commands fails, and GNU parallel prints: parallel: This job failed: Is there any…
Saurabh Nanda
  • 6,373
  • 5
  • 31
  • 60
3
votes
1 answer

Gnu parallel getting stuck, not giving output

Parallel seems to be installed properly as I am able to run basic parallel check commands. # parallel --version GNU parallel 20161222 Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016 Ole Tange and Free Software Foundation,…
akshay naidu
  • 115
  • 4
  • 18
3
votes
2 answers

Looking for an AWK alternative to speed up this command

I have been using AWK for this command but it is so slow. There must be a faster way to process this list from an aws s3 ls command s5cmd ls s3://bucket-name/* | awk -v AWS_BUCKET="bucket-name" '{cmd="aws s3api put-object-acl…
Joshua G. Edwards
  • 643
  • 6
  • 22
3
votes
1 answer

Matching output from functions running in parallel in Bash

My Bash script looks roughly like this: #!/bin/bash function a { echo -n "A is running..."; sleep 5; echo "done (A)" } function b { echo -n "B is running..."; sleep 1; echo "done (B)" } function c { echo -n "C is running..."; sleep 3; echo "done…
Max Power
  • 952
  • 9
  • 24
3
votes
2 answers

MV command line not working for certain files (OSX)

I downloaded a bunch of youtube videos and transcripts, and I am trying to rename them using the 11-character youtube ID that is part of the file title. I created a text file where I have on each line the old title, the new title and separated by…
user1029296
  • 609
  • 8
  • 17
3
votes
1 answer

GNU Parallel: --line-buffer option doesn't work with --pipe

Im trying to parallelize some tasks that needs to be processed on real time, so i was using --line-buffer. I was processing very long strings, but then i noticed that sometimes it hits the line lenght limit, making a command line too long error, so…
Andre Augusto
  • 84
  • 1
  • 8
3
votes
2 answers

Force gnu-parallel to treat replacement string as command

I want to pass a file containing a list of commands to gnu-parallel while using replacement strings (e.g.: {%}). Unfortunately, if replacement strings are used, gnu-parallel interprets the commands in the file as arguments to /bin/bash. Here's what…
diogom
  • 133
  • 4
1 2 3
…
48 49