Bash Script Loop Out of Memory?

Question

In bash I need to run a script that loops from i=1 to i=99999999 but it always run out of memory. Is there any workaround? or is there a max value for i?

first=1
last=99999999
randomString="CXCXQOOPSOIS"

for val in $( seq $first $last )
do
  padVal=$( printf "%010d\n" $val )
  hash=$( echo -n $randomString$padVal | md5sum )
  if [[ "$hash" =~ ^000000) ]]; then
    echo "Number: $val"  >> log_000000
    echo "$val added to log - please check."
  fi
done

It seems that loop allocates a new memory each cycle and overall allocated memory size reached a system limit, it is not possible giving you a concrete advice without seeing a loop code. — sll, Jan 09 '12 at 15:06
for the values of i, i actually append it to a string and then md5 hash it. when a hash comes up with say ^123123, i want to break from the loop. — John Marston, Jan 09 '12 at 15:14
And you didn't think your problem was the size of the string, as opposed to the loop? — Scott Hunter, Jan 09 '12 at 15:19
not very sure as i am new to shell programming... because my friend can loop till 12039023 but for some reason i cant.. he coded in python tho. and the length of his string and mine is the same... — John Marston, Jan 09 '12 at 15:23
I assumed he meant he was adding each number to a fixed string. I.e. "foo123", "foo124", "foo125". Password cracking, perhaps? — ghoti, Jan 09 '12 at 15:24
If we could see the code that was failing, it would be way easier to debug. — ghoti, Jan 09 '12 at 15:25

score 5 · Accepted Answer · answered Jan 09 '12 at 16:23

5

bash provides C-like syntax for loop:

first=1
last=99999999
randomString="CXCXQOOPSOIS"

for ((val=$first; val<$last; val++))
do
  padVal=$( printf "%010d\n" $val )
  hash=$( echo -n $randomString$padVal | md5sum )
  if [[ "$hash" =~ ^000000) ]]; then
    echo "Number: $val"  >> log_000000
    echo "$val added to log - please check."
  fi
done

answered Jan 09 '12 at 16:23

oHo

51,447
27
165
200

will this be better, memory wise? – John Marston Jan 09 '12 at 16:31
1

Yes, vastly better; it generates one number at a time, not 100 million up front. – Jonathan Leffler Jan 09 '12 at 16:38
thanks for the advice. just started shell scripting and having quite a bit of fun! – John Marston Jan 09 '12 at 16:43

score 1 · Answer 2 · answered Jan 09 '12 at 15:54

1

Your seq command generates 100 million numbers (bar a couple) and requires 800 MiB or so of memory to hold just the list of digits (probably an under-estimate; each number might be held in a separate memory allocation, which might mean 8 bytes for a pointer and 16 bytes for the allocated block, which triples the storage space estimate).

You can improve things dramatically by using:

for millions in $(seq 0 99)
do
    for smallstuff in $(seq -f "%6.0f" 0 999999)
    do
        val="$millions$smallstuff"
        ...
    done
done

This dramatically reduces the amount of memory needed; the only issue to watch is that it tests 0 which your original code did not.

answered Jan 09 '12 at 15:54

Jonathan Leffler

730,956
141
904
1,278

what does the line for millions in $(seq 0 99) do? – John Marston Jan 09 '12 at 16:02
It generates strings 0 to 99 (one hundred strings of up to 2 characters each). Actually, you probably want to use: `for millions in "" $(seq 1 99)` to generate an empty prefix. Also, the inner loop generates leading zeroes on the small stuff; if that matters, you have to be a bit more careful, but the basic idea of breaking up the loop so that you don't generate 100 million strings at the start still stands. – Jonathan Leffler Jan 09 '12 at 16:37

score 1 · Answer 3 · answered Jan 09 '12 at 16:30

1

If you still want to use seq => therefore separate seq and the loop using a pipe: |
This solution is more portable and can be used on other shells.
The memory print is still reduced, but this script requires to process two threads.

first=1
last=99999999
randomString="CXCXQOOPSOIS"

seq $first $last |
while read val
do
  padVal=$( printf "%010d\n" $val )
  hash=$( echo -n $randomString$padVal | md5sum )
  if [[ "$hash" =~ ^000000) ]]; then
    echo "Number: $val"  >> log_000000
    echo "$val added to log - please check."
  fi
done

answered Jan 09 '12 at 16:30

oHo

51,447
27
165
200

i don't necessarily need to use seq, will the c like for loop be a better option? – John Marston Jan 09 '12 at 16:35
The C loop is probably the best option; this is a valid alternative. Both are better than my solution, though my diagnosis of the cause of the trouble stands. – Jonathan Leffler Jan 09 '12 at 16:39

Bash Script Loop Out of Memory?

3 Answers3

Linked