how to speed up openssl generate md5 checksum

Question

I'am facing a problem, in AIX platform, we use a command to generate checksum:

Sample:

exec 0<list
while read line 
do
openssl md5 $line >> checksum.out
done

But this last for a long time. I find out that our cpus still have free resources. It's the openssl md5 running multithread? If not how can I let it run by multithread, or using other method to speed up it.

Best Regards

Void

score 0 · Answer 1 · edited May 23 '17 at 12:14

If I understand correctly from the answer and comments of this question, it can't be done as there are dependencies between the steps in the hashing algorithm (and I guess OpenSSL would have a multithreaded implementation if it was generally possible).

However you could always parallelize the tasks by starting n instances of openssl md5 in parallel.

For example (assuming n = 4 threads)

while read line; do
  openssl md5 $line >> checksum.out0 &
  openssl md5 $(read) >> checksum.out1 &
  openssl md5 $(read) >> checksum.out2 &
  openssl md5 $(read) >> checksum.out3
done

The last one should not run in the background if you want to keep the exact number of threads running at the same time. Also you may want to make sure that the different lines take about the same time to complete so you don't end up with race conditions.

Also this example is not really tested (using $(read)), and there are probably better ways to do it (for example let each instance write its output to a separate file and then concatenate them all afterwards - e.g. cat checksum.out* > checksum.out), but it should be enough of an idea to help you get started.

EDIT: I just tested and read works the way I hoped, so by making a new output file for each instance of openssl md5 with incremented numbers at the end (for example by including a counter variable) you can just add an extra line at the end of the script to cat the outputs into a single file.

Resulting script:

exec 0<list
COUNT=0
while read line; do
  openssl md5 $line >> checksum.out$((COUNT++)) &
  openssl md5 $(read) >> checksum.out$((COUNT++)) &
  openssl md5 $(read) >> checksum.out$((COUNT++)) &
  openssl md5 $(read) >> checksum.out$((COUNT++))
done
cat checksum.out* > checksum.out

Should do the trick (just remember to clean up all the temporary files afterwards...)

Thank you. You're correct. This should be the problem of multi-thread supporting. But I've changed the job for months... So cannot test it again because I don't have the machine... — krad, Oct 23 '15 at 15:34

how to speed up openssl generate md5 checksum

1 Answers1