6

I am trying to parallelize particle simulations with different parameters to save some time. Therefore I wanted to use GNUparallel to run a bash script for the different parameters. The script reads a file and then performs the simulation eg :

$bash script <<< input file

However:-

$cd ~/parameter_files ls | parallel bash script <<< {}

does not work at all. I am a total newbie to Linux and GNUparallel, so hopefully someone can help.

Abhijit Pritam Dutta
  • 5,521
  • 2
  • 11
  • 17
Physicus
  • 61
  • 1
  • 3

2 Answers2

5

You're almost right, use double quotes around the command

ls | parallel "bash script <<< {}"

Otherwise the here string <<< would feed the input into the parallel command rather than each individual bash script commands

I find this use case to be quite unusual however, since it means that basically your script is reading the filename string. If you just want to pass the files as arguments to the script, you can use

ls | parallel bash script

or if you want to pass the content of the files using standard input

ls | parallel "bash script < {}"
etopylight
  • 1,239
  • 1
  • 10
  • 15
  • The point is that Im reading the filename and then pass it as an argument to a python code. On the other hand I also need specific lines of the file besides its name to name my output variables. – Physicus Nov 17 '17 at 11:47
  • @Physicus I see, so I assume that your python code instead of bash would do the file I/O stuff based on the given filename – etopylight Nov 17 '17 at 13:09
0

You can use parallel like this on each file of ~/parameter_files directory:

find ~/parameter_files -maxdepth 1 -type f -print0 | parallel -0 bash ../script
  • find command finds all the files inside ~/parameter_files directory and returns them NUL terminated
  • parallel -0 bash ... feeds each filename to your shell script and executes it.
anubhava
  • 761,203
  • 64
  • 569
  • 643