Confusing results in benchmarking Java md5sum on different devices

Question

The short question:

Why does the calculation of the md5-sum of a 5 MB file in Java take 84 seconds on a Raspberry Pi, while a Mac needs only 25 ms?

The whole question:

I need to write a Java program, which calculates the md5- or sha-sum of a bunch of files, which have together the size of about 50 GB.

For this purpose I wrote a simple Java program, which calculates the checksum of a single 5 MB file. This is the Java program:

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.Arrays;

public class Main {

    public static final int BLOCKSIZE = 8*1024;

    public static void main(String[] args) throws FileNotFoundException, NoSuchAlgorithmException{
        String path = Main.class.getResource("file5M.img").getPath();
        File file = new File(path);
        FileInputStream fin = new FileInputStream(file);
        MessageDigest messageDigest = MessageDigest.getInstance("MD5");

        long fileSize = file.length();
        int length;
        long alreadyRead = 0;
        long startTime = System.currentTimeMillis();
        byte[] bytes = new byte[BLOCKSIZE];
        try {
            while (true) {
                int maxToRead = (int) (fileSize - alreadyRead < BLOCKSIZE ? fileSize - alreadyRead : BLOCKSIZE);

                if ((length = fin.read(bytes, 0, maxToRead)) < 0) break;
                messageDigest.update(bytes, 0, length);
                if ((alreadyRead += length) >= fileSize) break;
            }
        } catch (IOException ex){
            ex.printStackTrace();
        }
        byte[] md5 = messageDigest.digest();
        long stopTime = System.currentTimeMillis();
        long elapsedTime = stopTime - startTime;
        System.out.println("Time:\t" + elapsedTime + "\tRead:\t" + alreadyRead/1024/1024);
        System.out.println("MD5: " + Arrays.toString(md5));
    }
}

For creating a random fileimage, I used this Linux command:

dd if=/dev/urandom of=file5M.txt bs=1M count=5

Executing the program on different devices, lead to confusing results:

 <table style="width:100%">
  <tr>
    <th>Time in ms</th>
    <th>Computer</th>
    <th>CPU</th>
    <th>RAM</th>
    <th>Harddrive</th>
    <th>Operating-System</th>
  </tr>
  <tr>
    <td>24</td>
    <td>MacBook Pro (13-inch, 2016)</td>
    <td>3.3 GHz Intel Core i7</td>
    <td>8 GB 2133 MHz LPDDR3</td>
    <td>APPLE SSD AP1024J</td>
    <td>MacOs Sierra</td>
  </tr>
  <tr>
    <td>45000</td>
    <td>Raspberry Pi Modell B</td>
    <td>0.7 GHz ARMv6 (32-bit)</td>
    <td>256 MB</td>
    <td>PRO microSD Card (SD Adapter)</td>
    <td>Arch Linux</td>
  </tr>
  <tr>
    <td>7600</td>
    <td>Odroid XU4</td>
    <td>Exynos5 Octa Cortex™-A15 1.6Ghz quad core and Cortex™-A7 quad core CPUs</td>
    <td>2Gbyte LPDDR3 RAM PoP</td>
    <td>Samsung PRO microSD Card (SD Adapter)</td>
    <td>Arch Linux for Odroid-XU3</td>
  </tr>
  <tr>
    <td>300</td>
    <td>VirtualBox on MacBook Pro</td>
    <td>1 Core with 0.7GHz (21% of MacCPU) no PAE/NX, no acceleration</td>
    <td>256MB of MacRAM PIIX3 with APIC</td>
    <td>Dynamic Allocated 8GB (VDI)</td>
    <td>Arch Linux 64-Bit</td>
  </tr>
</table>

So why is the execution of the program so much faster on the MacBook, even if I restrict the CPU and the RAM in VirtualBox?

Where can the bottleneck be?

What do I have to do to make the program execute in about 300 ms on the Odroid-XU4?

Remarks:

I don't think it is the I/O of the microSD, because it reads the whole file very fast without calculating the md5sum.

Changing the cpu-frequency from 2Ghz to 500MHz on the odroid, increased computation time from 7 to 24 seconds.

Answer 1

The Raspberry Pi has a much lower RAM frequency than the MacBook. It's probably why it runs faster even in VirtualBox. Because when you read a file, it will be stored in RAM, and even if it's very fast, you have I/O access every time you read the file and sum it with MD5 algorithm.

Moreover, if you want to improve the performance, I suggest you to use threads in your program (dispatch the files between the threads). Note that the threads are useless if you only have one core like on your VM.