CryptoJS SHA256 Large File Progressive Checksum

Question

I am trying to use CryptoJS to get the checksum of large files( bigger than 500MB) so browser doesn't crash. I am already using a Web Worker with chunking. So, I am try to progressively use each chunk when I iterate through the chunks to update CryptoJS to start creating the checksum. However, it's still returning the wrong checksum at the end when I finalize. It's seems like it's only returning the checksum of the last chunk not the checksum of all the chunks. Can you let me know what I am doing wrong.

Also, I don't have to use CryptoJS as I find it to be slow but it seems the only library that can handle progressive encryption.

 var sha256 = CryptoJS.algo.SHA256.create(),
     sha256Update,
     checksum = [],
     chunker = function(workerData) {                            

       var file = workerData.fileBlob,
           totalFileSize = file.size,
           chunkLength = 3145728,
           start = 0,
           stop = chunkLength,
           i = 0, readSlicer,
           fileSlicer,
           chunk,
           chunkUint8,
           timerCounter = 0,
           hashConvert;

      var checker = function() {

          start = stop;
          stop += chunkLength;

          if(start >= totalFileSize) {
               console.log("Done reading file!", stop, totalFileSize);
               sha256Update.finalize();

               console.log("CheckSum : ", sha256Update._hash.toString(CryptoJS.enc.Hex));
               return;
                               }
               readBlock(start, chunkLength, file);
      };

      var readBlock = function(start, chunkLength, file) {

          readSlicer = new FileReaderSync();
          fileSlicer = file.slice(start, stop + 1);

          chunk = readSlicer.readAsArrayBuffer(fileSlicer);
          chunkUint8 = new Uint8Array(chunk);
          var wordArr = CryptoJS.lib.WordArray.create(chunkUint8);
              sha256Update = sha256.update(wordArr);
              checksum.push(sha256Update);
              checker();

      };

          readBlock(start, chunkLength, file);
   };

I haven't tried it, but it's probably because of the `+1` in the line `fileSlicer = file.slice(start, stop + 1);`. Remove it. — Artjom B., Apr 19 '15 at 16:27
@ArtjomB. Don't you hate when something is starring you in the face and you can't see it? That was it! Thanks! — Claude, Apr 19 '15 at 18:09
Yes, I hate this. I only noticed this, because I've written something like this for a SO question. — Artjom B., Apr 19 '15 at 18:10
Is your code working with the suggested fix of removing "+ 1" above? Were you able to optimize or reduce the code & make it adhere to typescript? — iammilind, Aug 24 '18 at 09:03

score 5 · Answer 1 · answered Jul 09 '20 at 21:35

I've cleaned up the above code a little; someone might find it useful (this is tested and works correctly):

import CryptoJS from 'crypto-js';


class ChecksumService {

  async sha256(file: File): Promise<string> {
    let sha256 = CryptoJS.algo.SHA256.create();
    const sliceSize = 10_485_760; // 10 MiB
    let start = 0;

    while (start < file.size) {
      const slice: Uint8Array = await this.readSlice(file, start, sliceSize);
      const wordArray = CryptoJS.lib.WordArray.create(slice);
      sha256 = sha256.update(wordArray);
      start += sliceSize;
    }

    sha256.finalize();

    return sha256._hash.toString();
  }

  private async readSlice(file: File, start: number, size: number): Promise<Uint8Array> {
    return new Promise<Uint8Array>((resolve, reject) => {
      const fileReader = new FileReader();
      const slice = file.slice(start, start + size);

      fileReader.onload = () => resolve(new Uint8Array(fileReader.result as any));
      fileReader.onerror = reject;
      fileReader.readAsArrayBuffer(slice);
    });
  }

}

Since the chunks are being read asynchronously, how do you know the hash is being calculated in the correct order? This article basically does the same, but has code to check that it's done in the right order: https://medium.com/@0xVaccaro/hashing-big-file-with-filereader-js-e0a5c898fc98 — Mischa, Jul 14 '21 at 12:57
it always awaits the next slice, so I don't think it can get out of order — Zoltan Tarcsay, Jul 15 '21 at 13:10
@ZoltanTarcsay, can you share the decrypt functionality for that — kumaran, Aug 31 '23 at 04:38

CryptoJS SHA256 Large File Progressive Checksum

1 Answers1