GNU Parallel - redirect output to a file with a specific name

Question

In bash I am running GnuPG to decrypt some files and I would like the output to be redirected to a file having the same name, but a different extension. Basically, if my file is named

file1.sc.xz.gpg

the file which comes out after running the GnuPG tool I would like to be stored inside another file called

file1.sc.xz

I am currently trying

find . -type f | parallel "gpg {} > {}.sc.xz"

but this results in a file called file1.sc.xz.gpg.sc.xz. How can I do this?

Later edit: I would like to do this inside one single bash command, without knowing the filename in advance.

If `parallel` is an essential part of your question, consider mentioning it in the title. It's not part of bash, and a surprisingly complex tool. — Charles Duffy, Jun 10 '15 at 15:21
(that said -- would an answer using `xargs -P` to work in parallel *without* GNU parallel be welcome?) — Charles Duffy, Jun 10 '15 at 15:23

Maxim Egorushkin · Answer 1 · 2015-06-10T15:29:01.930

6

You can use bash variable expansion to chop off the extension:

$ f=file1.sc.xz.gpg
$ echo ${f%.*}
file1.sc.xz

E.g.:

find . -type f | parallel bash -c 'f="{}"; g="${f%.*}"; gpg "$f" > "$g"'

Alternatively, use expansion of parallel:

find . -type f | parallel 'gpg "{}" > "{.}"'

edited Jun 10 '15 at 15:29

answered Jun 10 '15 at 15:16

Maxim Egorushkin

131,725
17
180
271

BTW, the answers using `find` and `parallel` here won't work safely with all possible filenames. Consider names with literal newlines for a particular interesting corner case. – Charles Duffy Jun 10 '15 at 15:24
...though given the lack of proper quoting, the current implementations are buggy even with files with simple whitespace in their names. – Charles Duffy Jun 10 '15 at 15:25
@CharlesDuffy I recently accidentally copied a folder with a trailing space on my Android phone. Could not move files from it or remove the directory from the phone itself. – Maxim Egorushkin Jun 10 '15 at 15:26
@MaximEgorushkin, ...if you want some guidance on dealing with those cases correctly, I'd be happy to help. Actually working on an answer demonstrating NUL-delimited streams of names right now. – Charles Duffy Jun 10 '15 at 15:26
5

Use `-print0` and `parallel --null` to safe-guard against hostile file names – Aaron Digulla Jun 10 '15 at 15:28
@CharlesDuffy I defer that to classic http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html – Maxim Egorushkin Jun 10 '15 at 15:28
1

@MaximEgorushkin, ...I actually disagree rather strongly with David there. Filenames with arbitrary content aren't broken; code that con't handle them correctly is broken. – Charles Duffy Jun 10 '15 at 15:29
@CharlesDuffy What about filenames with backspaces? – Maxim Egorushkin Jun 10 '15 at 15:30
@MaximEgorushkin, ...what about them? – Charles Duffy Jun 10 '15 at 15:30
Anyhow, we don't live in a world where the set of valid names has been restricted, even if that world would be a sensible or ideal one. Buffer overflows dumping junk into filenames *happens*; malicious users injecting content through filenames *happens*; and well-written code should handle all those cases. – Charles Duffy Jun 10 '15 at 15:33
(One of my former employers lost several TB of records used for accounting purposes on account of a buffer overflow throwing junk into a filename and a shell script that didn't quote correctly doing cleanup; it wasn't pretty). – Charles Duffy Jun 10 '15 at 15:35
@CharlesDuffy In ideal world, that is true. In a real world handling all possible characters in filenames may be economically unreasonable: the client may not wish to pay for an extra day of work to handle funny characters. – Maxim Egorushkin Jun 10 '15 at 15:35
The engineering time that former employer spent cleaning up after the mess I mention in the comment above far exceeded what prevention would have run. – Charles Duffy Jun 10 '15 at 15:36
@MaximEgorushkin: When trying to run find . -type f | parallel bash -c 'f="{}"; g="${f%.*}"; gpg "$f" > "$g"' I get /usr/local/bin/bash: : No such file or directory – Crista23 Jun 10 '15 at 15:37
@MaximEgorushkin I cannot get any output file if I run this: find . -type f | parallel bash -c 'f="{}"; g="${f%.*}"; gpg "$f" > "$g"' Which is the extra stuff? – Crista23 Jun 10 '15 at 15:42
Insert `echo` in front of `gpg` to see what it is trying to invoke. – Maxim Egorushkin Jun 10 '15 at 15:44
Also, substituting names into code here, you have a security risk if any filename contains, say, the literal string `$(rm -rf .)`. Much safer to keep data and code separately, pulling arguments off the command line in your script rather than substituting into it. – Charles Duffy Jun 10 '15 at 15:45

score 3 · Answer 2 · answered Jun 10 '15 at 22:42

If file names are guaranteed not to contain \n:

find . -type f | parallel gpg {} '>' {.}
parallel gpg {} '>' {.} ::: *

If file names may contain \n:

find . -type f -print0 | parallel -0 gpg {} '>' {.}
parallel -0 gpg {} '>' {.} ::: *

Note that opposite shell variables GNU Parallel's substitution strings should not be quoted. This will not create the file 12", but instead 12\" (which is wrong):

parallel "touch '{}'" ::: '12"'

These will all do the right thing:

parallel touch '{}' ::: '12"'
parallel "touch {}" ::: '12"'
parallel touch {} ::: '12"'

score 0 · Answer 3 · answered Jun 10 '15 at 15:28

0

find . -type f -print0 | \
  xargs -P 0 -n 1 -0 \
    bash -s -c 'for f; do g=${f%.*}; gpg "$f" >"$g"; done' _

Right now this processes only one file per shell, but you could trivially modify that by changing the -n value.

answered Jun 10 '15 at 15:28

Charles Duffy

280,126
43
390
441

Don't you need double quotes around `${f%.*}`? – Maxim Egorushkin Jun 10 '15 at 15:43
1

@MaximEgorushkin, no; string-splitting and glob expansion don't happen on scalar assignments. – Charles Duffy Jun 10 '15 at 15:43

GNU Parallel - redirect output to a file with a specific name

3 Answers3