3

I have a long list of filenames in filenames.txt file. These files are lzo compressed and I use lzop to decompress them for further processing in a pipeline.

cat filenames.txt | (xargs lzop -dc || true) | python lineprocessor.py  > output.txt

So filenames are input to the lzop -dc line by line. Then they are decompressed and piped into the lineprocessor.py script that I have written. Finally the output of lineprocessor.py is written in output.txt.

The problem is that some files in filenames.txt are not properly compressed and lzop crashes and so does the whole pipeline. I added the || true to prevent this but it didn't help. lzop doesn't have the option to ignore the error. I don't care about the incorrectly compressed files.

Is there any way I can work around this problem easily? I want the pipeline to continue no matter what happens to lzop -dc command.

Ash
  • 3,428
  • 1
  • 34
  • 44

1 Answers1

3
while read filename; do
    lzop -fdc "$filename" | python lineprocessor.py
done < filenames.txt >> output.txt
Vladimir Kunschikov
  • 1,735
  • 1
  • 16
  • 16
  • 1
    While this code snippet may solve the question, [including an explanation](http://meta.stackexchange.com/questions/114762/explaining-entirely-‌​code-based-answers) really helps to improve the quality of your post. Remember that you are answering the question for readers in the future, and those people might not know the reasons for your code suggestion. – NathanOliver Dec 21 '15 at 19:48
  • 1
    @vladimirkunschikov, I tried to edit the script to use `"$filename"` instead of `$filename` but couldn't (at least six chars of edition needed). That can make the script resistant to filenames with embedded spaces. Please, edit (you can) – Luis Colorado Dec 22 '15 at 09:26