14

This question is related to How to use grep efficiently?

I am trying to search for a "string" in a folder which has 8-10 million small (~2-3kb) plain text files. I need to know all the files which has "string".

At first I used this

grep "string"

That was super slow.

Then I tried

grep * "string" {} \; -print

Based on linked question, I used this 

 find . | xargs -0 -n1 -P8 grep -H "string"

I get this error:

xargs: argument line too long

Does anyone know a way to accomplish this task relatively quicker?

I run this search on a server machine which has more than 50GB of available RAM, and 14 cores of CPU. I wish somehow I could use all that processing power to run this search faster.

Community
  • 1
  • 1
Watt
  • 3,118
  • 14
  • 54
  • 85

4 Answers4

13

You should remove -0 argument to xargs and up -n parameter instead:

... | xargs -n16 ...
Alexander L. Belikoff
  • 5,698
  • 1
  • 25
  • 31
11

It's not that big stack of files (kudos to 10⁷ files - a messys dream) but I created 100k files (400 MB overall) with 

for i in {1..100000}; do head -c 10 /dev/urandom > dummy_$i; done

and made some tests for pure curiosity (the keyword 10 I was searching is chosen randomly):

> time find . | xargs -n1 -P8 grep -H "10"
real 0m22.626s
user 0m0.572s
sys  0m5.800s

> time find . | xargs -n8 -P8 grep -H "10"
real 0m3.195s
user 0m0.180s
sys  0m0.748s

> time grep "10" *
real 0m0.879s
user 0m0.512s
sys  0m0.328s

> time awk '/10/' *
real 0m1.123s
user 0m0.760s
sys  0m0.348s

> time sed -n '/10/p' *
real 0m1.531s
user 0m0.896s
sys  0m0.616s

> time perl -ne 'print if /10/' *
real 0m1.428s
user 0m1.004s
sys  0m0.408s

Btw. there isn't a big difference in running time if I suppress the output with piping STDOUT to /dev/null. I am using Ubuntu 12.04 on a not so powerful laptop ;) My CPU is Intel(R) Core(TM) i3-3110M CPU @ 2.40GHz.

More curiosity:

> time find . | xargs -n1 -P8 grep -H "10" 1>/dev/null

real 0m22.590s
user 0m0.616s
sys  0m5.876s

> time find . | xargs -n4 -P8 grep -H "10" 1>/dev/null

real m5.604s
user 0m0.196s
sys  0m1.488s

> time find . | xargs -n8 -P8 grep -H "10" 1>/dev/null

real 0m2.939s
user 0m0.140s
sys  0m0.784s

> time find . | xargs -n16 -P8 grep -H "10" 1>/dev/null

real 0m1.574s
user 0m0.108s
sys  0m0.428s

> time find . | xargs -n32 -P8 grep -H "10" 1>/dev/null

real 0m0.907s
user 0m0.084s
sys  0m0.264s

> time find . | xargs -n1024 -P8 grep -H "10" 1>/dev/null

real 0m0.245s
user 0m0.136s
sys  0m0.404s

> time find . | xargs -n100000 -P8 grep -H "10" 1>/dev/null

real 0m0.224s
user 0m0.100s
sys  0m0.520s
EverythingRightPlace
  • 1,197
  • 12
  • 33
-1

8 million files is a lot in one directory! However, 8 million times 2kb is 16GB and you have 50GB of RAM. I am thinking of a RAMdisk...

Mark Setchell
  • 191,897
  • 31
  • 273
  • 432
  • Yes, it is too many files in a folder. Can you please elaborate your solution on how search the "string" faster? – Watt Oct 30 '13 at 22:03
  • Sadly I don't have a solution yet... I am still trying to understand the parameters of the question. What OS are you using? What filesystem are you using? Have you tried running the "find" command on its own and timing it? time find . | wc -l – Mark Setchell Oct 30 '13 at 22:12
  • OS: Ubuntu (latest version) . Find returned result within 2 seconds – Watt Oct 30 '13 at 22:15
-3

If you've got that much RAM, why not read it all into memory and use a regular expression library to search? It's a simple C program:

    #include <fcntl.h>
    #include <regex.h>
    ...
Anonymous
  • 11
  • 2