Is it possible to parallelize this loop in python?

Question

I've been working on a program to obtain a rainbow table, using a crc32 hash.

The main loop of the program is the following:

from zlib import crc32
from string import ascii_lowercase
from itertools import product

...

rt = {}
i = 0
for p in product(ascii_lowercase, repeat = 8):
    i += 1
    print('\n%d' % i)
    p = ''.join(p)
    print('\nCurrent password = %s' % p)
    r = bytes(p, 'utf-8')
    h = hex(crc32(r))
    for j in range(2**20):
        h = list(h)
        del h[0]
        del h[0]
        for k in range(0, len(h)):
            if (h[k].isdigit()):
                h[k] = chr(ord(h[k]) + 50)
        h = ''.join(h)
        r = bytes(h, 'utf-8')
        h = hex(crc32(r))
    h = list(h)
    del h[0]
    del h[0]
    h = ''.join(h)
    print('\nFinal hash = %s\n' % h)
    rt[p] = h
    if (i == 2**20):
        break

So, the code does its works as I intend it to, and when it exits the loop it stores the generated rainbow table (variable rt) into memory. Well, with the current number of iterations shown in the above code, it would take several days to complete its execution, and I need to create this table, as well as others with different iterations through the loops, in order to do some tests on them.

I thought that I'd be a good idea to attempt to parallelize it, but after going through the documentation on multiprocessing and lurking through some posts about it, I'm still unable to parallelize it in a correct way.

Thanks in advance!

Place content of outmost for-loop into a function (except the first and the last three lines) taking `p` and returning `(p, h)`. Create a `Pool` and use its `map_async` or `imap_unordered` with the `product(...)` as input iterable to the newly create function. — Michael Butscher, Oct 16 '18 at 13:29
I'm sorry Michael, but following your recommendation I could only come up with the following: `with Pool(4) as pool: print(pool.map_async(table, (product(ascii_lowercase, repeat = 8)))) ` Where table is the function which you suggested. Doing this I've only managed to almost crash my machine, so I'd ask you to please show me a code example of your suggestion. Thank you. — Atalajaka, Oct 17 '18 at 07:27

score 1 · Accepted Answer · answered Oct 17 '18 at 13:33

Try this:

from zlib import crc32
from string import ascii_lowercase
from itertools import product
from multiprocessing.pool import Pool

def table(p):
    p = ''.join(p)
    r = bytes(p, 'ascii')
    h = hex(crc32(r))
    for j in range(2**20):
        h = list(h)
        del h[0]
        del h[0]
        for k in range(0, len(h)):
            if (h[k].isdigit()):
                h[k] = chr(ord(h[k]) + 50)
        h = ''.join(h)
        r = bytes(h, 'ascii')
        h = hex(crc32(r))
    h = list(h)
    del h[0]
    del h[0]
    h = ''.join(h)

    return (p, h)

if __name__ == "__main__":
    rt = {}
    i = 0
    with Pool(4) as pool:
        for p, h in pool.imap_unordered(table, product(ascii_lowercase, repeat = 8)):
            print('\n%d' % i)
            print('\nCurrent password = %s' % p)
            print('\nFinal hash = %s\n' % h)
            i += 1
            rt[p] = h
            if i == 2**20:
                break

Be aware that the final break condition i == 2**20 may work inexactly because of the arbitrary order of the calculation tasks.

It works perfectly and so much quicker, thank you very much! My mistake was in how I passed the arguments in the pool.imap_unordered. I really appreciate you help :) — Atalajaka, Oct 18 '18 at 14:23

Is it possible to parallelize this loop in python?

1 Answers1

Linked