Memory problems with xargs and grep and pattern from file

Asked Jan 30 '13 at 15:35

Active Jan 30 '13 at 15:35

Viewed 279 times

Within a makefile I run the following command

find SOURCE_DIR -name '*.gz' | xargs -P4 -L1 bash -c 'zcat $$1 | grep -F -f <(zcat patternfile.csv.gz) | gzip > TARGET_DIR/$${1##*/}' -

patternfile.csv.gz contains 2M entries with an unzipped file size of 100MB, each file in SOURCE_DIR has a zipped file size of ~20MB.

However, each xargs process consumes more than 6GB of RAM. Does this make sense or do I miss something here?

Thanks for your help.

asked Jan 30 '13 at 15:35

Mario Konschake

2

I suspect that `bash` construct is not operating as you think it is. Change `xargs` to `xargs -t` and change `bash` to `bash -x` to verify that the commands you are running are what you expect. In particular, the `$$1` constructs are doing something I don't think is what you expect... – twalberg Jan 30 '13 at 16:19
I just learned that automata are not linear in memory, was not aware of this. – Mario Konschake Jan 31 '13 at 08:27

0 Answers0