Shell TAR files from list or using pipe

Question

I was trying to pipe a TAR after listing a bunch of files. First I tried to list my files according to their date. Like this:

ll /home/dan/test/dir*/file* | grep 2014
-rw-r--r-- 1 root root 0 Jan  2  2014 /home/dan/test/dir1/file1_2014_1
-rw-r--r-- 1 root root 0 Jan  2  2014 /home/dan/test/dir1/file1_2014_10
-rw-r--r-- 1 root root 0 Jan  2  2014 /home/dan/test/dir1/file1_2014_100
-rw-r--r-- 1 root root 0 Jan  2  2014 /home/dan/test/dir1/file1_2014_11
-rw-r--r-- 1 root root 0 Jan  2  2014 /home/dan/test/dir1/file1_2014_12
...
-rw-r--r-- 1 root root 0 Jan  2  2014 /home/dan/test/dir4/file4_2014_97
-rw-r--r-- 1 root root 0 Jan  2  2014 /home/dan/test/dir4/file4_2014_98
-rw-r--r-- 1 root root 0 Jan  2  2014 /home/dan/test/dir4/file4_2014_99

Since the result is a bit poluted, I tried to filter it.

ll /home/dan/test/dir*/file* | grep 2014 | cut -f 2- -d "/" | sed 's/home/\/home/'
/home/dan/test/dir1/file1_2014_1
/home/dan/test/dir1/file1_2014_10
/home/dan/test/dir1/file1_2014_100
/home/dan/test/dir1/file1_2014_11
/home/dan/test/dir1/file1_2014_12
...
/home/dan/test/dir4/file4_2014_97
/home/dan/test/dir4/file4_2014_98
/home/dan/test/dir4/file4_2014_99

which gave just file paths and names. Then I tried to TAR this list using pipe but didn't work out.

So I exported the list to a file

ll /home/dan/test/dir*/file* | grep 2014 | cut -f 2- -d "/" | sed 's/home/\/home/' > list_files_2014.txt

Now, I need to TAR this list.

But when I try it gives me an error:

tar -cvjf test.tar.bz2 -T list_files_2014.txt
tar: Removing leading `/' from member names
/home/dan/test/dir1/file1_2014_1
/home/dan/test/dir1/file1_2014_10
/home/dan/test/dir1/file1_2014_100
/bin/sh: 1: /home/dan/test/dir1/file1_2014_11
bzip2: not found/home/dan/test/dir1/file1_2014_12

/home/dan/test/dir1/file1_2014_13
/home/dan/test/dir1/file1_2014_14
/home/dan/test/dir1/file1_2014_15
/home/dan/test/dir1/file1_2014_16
/home/dan/test/dir1/file1_2014_17
/home/dan/test/dir1/file1_2014_18
/home/dan/test/dir1/file1_2014_19
/home/dan/test/dir1/file1_2014_2
/home/dan/test/dir1/file1_2014_20
/home/dan/test/dir1/file1_2014_21
/home/dan/test/dir1/file1_2014_22
/home/dan/test/dir1/file1_2014_23
/home/dan/test/dir1/file1_2014_24
/home/dan/test/dir1/file1_2014_25
/home/dan/test/dir1/file1_2014_26
tar: test.tar.bz2: Cannot write: Broken pipe
tar: Child returned status 127
tar: Error is not recoverable: exiting now

Can anyone help me out? Thanks

[Why *not* parse `ls`?](http://unix.stackexchange.com/questions/128985/why-not-parse-ls) — Cyrus, Mar 09 '20 at 23:50
`bzip2` was not found. Either install it or don't use `-j` option to `tar` — jhnc, Mar 10 '20 at 03:41
The main issue is that you need to install bzip2; however, with respect to replacing the `ls | grep` antipattern -- do you have the GNU version of `find`? (If so, you can use a `-newerXY` primitive, like `-newermt`, to find files newer or older than a specific time). — Charles Duffy, Mar 10 '20 at 16:05
(btw, bz2 isn't really a good choice these days -- there are algorithms that are faster at the same compression ratio, or which compress much better at the same performance). — Charles Duffy, Mar 10 '20 at 16:06

score 1 · Answer 1 · answered Mar 10 '20 at 16:00

You should not parse ls because the output of ls is not stable and only meant to be read by humans. After an update of the OS this output (its format) can change, rendering your script buggy. The link @Cyrus provided leads to a way more exhaustive explanation which I don't want to repeat here. Anyway, it is way better to use other Unix tools to achieve your goals, e.g. find.

But this is not your issue here.

Your issue is merely that your tar tries to spawn a second process for the compression you have chosen. It tries to execute a program called bzip2. This doesn't seem to be installed on your computer, so it fails. The message comes a little delayed, because first the second process is created, and this parallel running other process then tries to run the program bzip2. In the meantime the tar process already started packaging the files (and made some log messages). Since the two processes are connected by a pipe, the first process finally receives a SIGPIPE when it tries to write into this pipe (which now lacks the reader process).

So you either have to install bzip2 or you just choose another compression algorithm which already is installed. You could choose gzip by replacing the option -j by the option -z, for example. The man page of tar names a bunch of other supported compression algorithms.

Shell TAR files from list or using pipe

1 Answers1