Read and Write to File Chunk By Chunk

Question

I am developing a file encryption program. I was using the function below to encrypt files until I realized that it is not suitable for big ones; because it reads all file content into memory. Now, I need to create a function that can read and write file content in chunks. How can I do this?

private fun encryptFile(file: File) {
   val originalData = file.readBytes()
   val encryptData = encrypt(originalData)
   encryptData?.run {
       file.writeBytes(this)
   }
}

@Andreas No, it just gets a ByteArray, encrypts it, and returns a new ByteArray. — dreyzz, Dec 11 '20 at 21:14
A lot of file encryption programs run with CipherOutputStreams as they support buffered encryption ("chunks"). Search for e.g. "Java CipherOutputStream AES CBC" — Michael Fehr, Dec 11 '20 at 21:34

score 1 · Answer 1 · answered Dec 11 '20 at 21:32

1

Your encrypt function obviously can't stay that way. It'll have to become a thing that wraps an InputStream or OutputStream, and then it's fairly trivial.

Note that handrolling encryption is a near 100% guarantee you'll mess it up, and crypto streams already exist. Any reason you're reinventing a wheel and signing up to mess up security by reinventing things you shouldn't?

answered Dec 11 '20 at 21:32

rzwitserloot

85,357
5
51
72

Can you give sample code about my encryption function? – dreyzz Dec 11 '20 at 21:53
The accepted answer of [This question](https://stackoverflow.com/questions/41413439/encrypting-and-decrypting-a-file-using-cipherinputstream-and-cipheroutputstream) for example. – rzwitserloot Dec 12 '20 at 03:15

Co ti · Answer 2 · 2020-12-11T21:36:56.877

Have a look at code. OP

    // ...
StringBuilder sb = new StringBuilder(); 
String line;
while ((line = inputStream.readLine()) != null) {
    sb.append(line);

    // if enough content is read, extract the chunk
    while (sb.length() >= chunkSize) {

        String c = sb.substring(0, chunkSize);
        // do something with the string

        // add the remaining content to the next chunk
        sb = new StringBuilder(sb.substring(chunkSize));
    }
}
// thats the last chunk
String c = sb.toString();
// do something with the string

EDIT: What about using Chilkat library link to download a Chillkat lib

Code example for encypting chunk of file

import com.chilkatsoft.*;

public class ChilkatExample {

  static {
    try {
        System.loadLibrary("chilkat");
    } catch (UnsatisfiedLinkError e) {
      System.err.println("Native code library failed to load.\n" + e);
      System.exit(1);
    }
  }

  public static void main(String argv[])
  {

CkCrypt2 crypt = new CkCrypt2();

crypt.put_CryptAlgorithm("aes");
crypt.put_CipherMode("cbc");
crypt.put_KeyLength(128);

crypt.SetEncodedKey("000102030405060708090A0B0C0D0E0F","hex");
crypt.SetEncodedIV("000102030405060708090A0B0C0D0E0F","hex");

String fileToEncrypt = "qa_data/hamlet.xml";
CkFileAccess facIn = new CkFileAccess();
boolean success = facIn.OpenForRead(fileToEncrypt);
if (success != true) {
    System.out.println("Failed to open file that is to be encrytped.");
    return;
    }

String outputEncryptedFile = "qa_output/hamlet.enc";
CkFileAccess facOutEnc = new CkFileAccess();
success = facOutEnc.OpenForWrite(outputEncryptedFile);
if (success != true) {
    System.out.println("Failed to encrypted output file.");
    return;
    }

// Let's encrypt in 10000 byte chunks.
int chunkSize = 10000;
int numChunks = facIn.GetNumBlocks(chunkSize);

crypt.put_FirstChunk(true);
crypt.put_LastChunk(false);

CkBinData bd = new CkBinData();

int i = 0;
while (i < numChunks) {
    i = i+1;
    if (i == numChunks) {
        crypt.put_LastChunk(true);
        }

    // Read the next chunk from the file.
    // The last chunk will be whatever amount remains in the file..
    bd.Clear();
    facIn.FileReadBd(chunkSize,bd);

    // Encrypt.
    crypt.EncryptBd(bd);

    // Write the encrypted chunk to the output file.
    facOutEnc.FileWriteBd(bd,0,0);

    crypt.put_FirstChunk(false);
    }

// Make sure both FirstChunk and LastChunk are restored to true after
// encrypting or decrypting in chunks.  Otherwise subsequent encryptions/decryptions
// will produce unexpected results.
crypt.put_FirstChunk(true);
crypt.put_LastChunk(true);

facIn.FileClose();
facOutEnc.FileClose();

// Decrypt the encrypted output file in a single call using CBC mode:
String decryptedFile = "qa_output/hamlet_dec.xml";
success = crypt.CkDecryptFile(outputEncryptedFile,decryptedFile);
// Assume success for the example..

// Compare the contents of the decrypted file with the original file:
boolean bSame = facIn.FileContentsEqual(fileToEncrypt,decryptedFile);
System.out.println("bSame = " + bSame);
  }
}

Thanks but there is no file writing logic in this code. And I want to read and write in Byte arrays because the files I am handling are binary. Also my encryption function uses ByteArray, not String. I do not want to convert String <=> ByteArray because this can corrupt file content. — dreyzz, Dec 11 '20 at 21:28
Also, StringBuilder is strings/chars, and files are bytes, so this answer is not at all a good idea — rzwitserloot, Dec 11 '20 at 21:30

Read and Write to File Chunk By Chunk

2 Answers2