SHA256 of data stream

Question

I am having a use case where i am downloading the file from URL using GET request. Is it possible to calculate SHA256 of file stream without saving to disk or holding an entire object in memory?

Answer 1

This can be accomplished with a MessageDigest and Sink.fold.

First we need a function to create an empty digest and a function to update a digest with a ByteBuffer:

import java.security.MessageDigest
import java.nio.ByteBuffer

def emptySHA256Digest : MessageDigest = MessageDigest getInstance "SHA-256" 

val updateDigest : (MessageDigest, ByteBuffer) => MessageDigest = 
  (messageDigest, byteBuffer) => {
    messageDigest update byteBuffer
    messageDigest
  }

These two functions can then be used within a fold which is applied to the entity of an HttpResponse to update the digest with all ByteString values in the entity:

import akka.http.scaladsl.model.HttpResponse

val responseBodyToDigest : HttpResponse => Future[MessageDigest] = 
  (_ : HttpResponse)
    .entity
    .dataBytes
    .map(_.asByteBuffer)
    .runFold(emptySHA256Digest)(updateDigest)

Answer 2

You would need Flow which will transform one data to another data. In your case, you want to transform plain text to sha256 text.

def digest(algorithm: String = "SHA-256"): Flow[ByteString, ByteString, NotUsed] = {
    Flow[ByteString].fold(MessageDigest.getInstance(algorithm)) {
        case (digest: MessageDigest, bytes:ByteString) => {digest.update(bytes.asByteBuffer); digest}}
          .map {case md: MessageDigest => ByteString(md.digest())}
}

Answer 3

Use one of MessageDigest.getInstance("SHA-256").update methods.

Pass your file piece-by-piece to it.

Then call digest()

Sample code:

    MessageDigest instance = MessageDigest.getInstance("SHA-256");
    for(byte[] arr: source){
        instance.update(arr);
    }
    byte[] result = instance.digest();