How to effeciently read chunk of bytes of a given range from a large encrypted file in java?

Question

I have a large encrypted file(10GB+) in server. I need to transfer the decrypted file to the client in small chunks. When a client make a request for a chunk of bytes (say 18 to 45) I have to random access the file, read the specific bytes, decrypt it and transfer it to the client using ServletResponseStream.

But since the file is encrypted I have to read the file as blocks of 16 bytes in order to decrypt correctly.

So if client requests to get from byte 18 to 45, in the server I have to read the file in multiples of 16 bytes block. So I have to random access the file from byte 16 to 48. Then decrypt it. After decryption I have to skip 2 bytes from the first and 3 bytes from the last to return the appropriate chunk of data client requested.

Here is what I am trying to do

Adjust start and end for encrypted files

long start = 15; // input from client
long end = 45; // input from client
long skipStart = 0; // need to skip for encrypted file
long skipEnd = 0;

// encrypted files, it must be access in blocks of 16 bytes
if(fileisEncrypted){
   skipStart = start % 16;  // skip 2 byte at start
   skipEnd = 16 - end % 16; // skip 3 byte at end
   start = start - skipStart; // start becomes 16
   end = end + skipEnd; // end becomes 48
}

Access the encrypted file data from start to end

try(final FileChannel channel = FileChannel.open(services.getPhysicalFile(datafile).toPath())){
    MappedByteBuffer mappedByteBuffer = channel.map(FileChannel.MapMode.READ_ONLY, start, end-start);

    // *** No idea how to convert MappedByteBuffer into input stream ***
    // InputStream is = (How do I get inputstream for byte 16 to 48 here?)

    // the medhod I used earlier to decrypt the all file atonce, now somehow I need the inputstream of specific range
    is = new FileEncryptionUtil().getCipherInputStream(is,
                        EncodeUtil.decodeSeedValue(encryptionKeyRef), AESCipher.DECRYPT_MODE);

    // transfering decrypted input stream to servlet response
    OutputStream outputStream = response.getOutputStream();
    // *** now for chunk transfer, here I also need to 
    //     skip 2 bytes at the start and 3 bytes from the end. 
    //     How to do it? ***/
    org.apache.commons.io.IOUtils.copy(is, outputStream)
}

I am missing few steps in the code given above. I know I could try to read byte by byte and the ignore 2byte from first and 3 byte from last. But I am not sure if it will be efficient enough. Moreover, the client could request a large chunk say from byte 18 to 2048 which would require to read and decrypt almost two gigabytes of data. I am afraid creating a large byte array will consume too much memory.

How can I efficiently do it without putting too much pressure on server processing or memory? Any ideas?

Answer 1

As you haven't specified which cipher mode you're using, I'll assume that you're using AES in CTR mode, as it's designed to read random chunks of big files without having to decrypt them completely.

With AES-CTR, you can stream the file through the decryption code and send the blocks back to the client as soon as they are available. So you only need a few arrays the size of the AES block in memory, all the rest is read from the disk. You would need to add special logic to skip some byes on the first and last block (but you don't need to load the whole thing in memory).

There's an example of how to do this in another SO question (this only performs the seek): Seeking in AES-CTR-encrypted input . After that you can skip the first few bytes, read until the last block and adjust that to the number of bytes your client requested.

Answer 2

After researching for awhile. This is how I solved it. First I created a ByteBufferInputStream class. To read from MappedByteBuffer

public class ByteBufferInputStream extends InputStream {
    private ByteBuffer byteBuffer;

    public ByteBufferInputStream () {
    }

    /** Creates a stream with a new non-direct buffer of the specified size. The position and limit of the buffer is zero. */
    public ByteBufferInputStream (int bufferSize) {
        this(ByteBuffer.allocate(bufferSize));
        byteBuffer.flip();
    }

    /** Creates an uninitialized stream that cannot be used until {@link #setByteBuffer(ByteBuffer)} is called. */
    public ByteBufferInputStream (ByteBuffer byteBuffer) {
        this.byteBuffer = byteBuffer;
    }

    public ByteBuffer getByteBuffer () {
        return byteBuffer;
    }

    public void setByteBuffer (ByteBuffer byteBuffer) {
        this.byteBuffer = byteBuffer;
    }

    public int read () throws IOException {
        if (!byteBuffer.hasRemaining()) return -1;
        return byteBuffer.get();
    }

    public int read (byte[] bytes, int offset, int length) throws IOException {
        int count = Math.min(byteBuffer.remaining(), length);
        if (count == 0) return -1;
        byteBuffer.get(bytes, offset, count);
        return count;
    }

    public int available () throws IOException {
        return byteBuffer.remaining();
    }
}

Then created BlockInputStream class by extending InputStream which will allow to skip the extra bytes and read internal input stream in multiples of 16 bytes block.

public class BlockInputStream extends InputStream {
    private final BufferedInputStream inputStream;
    private final long totalLength;
    private final long skip;
    private long read = 0;
    private byte[] buff = new byte[16];
    private ByteArrayInputStream blockInputStream;

    public BlockInputStream(InputStream inputStream, long skip, long length) throws IOException {
        this.inputStream = new BufferedInputStream(inputStream);
        this.skip = skip;
        this.totalLength = length + skip;
        if(skip > 0) {
            byte[] b = new byte[(int)skip];
            read(b);
            b = null;
        }
    }


    private int readBlock() throws IOException {
        int count = inputStream.read(buff);
        blockInputStream = new ByteArrayInputStream(buff);
        return count;
    }

    @Override
    public int read () throws IOException {
        byte[] b = new byte[1];
        read(b);
        return (int)b[1];
    }

    @Override
    public int read(byte[] b) throws IOException {
        return read(b, 0, b.length);
    }

    @Override
    public int read (byte[] bytes, int offset, int length) throws IOException {
        long remaining = totalLength - read;
        if(remaining < 1){
            return -1;
        }
        int bytesToRead = (int)Math.min(length, remaining);
        int n = 0;
        while(bytesToRead > 0){
            if(read % 16 == 0 && bytesToRead % 16 == 0){
                int count = inputStream.read(bytes, offset, bytesToRead);
                read += count;
                offset += count;
                bytesToRead -= count;
                n += count;
            } else {
                if(blockInputStream != null && blockInputStream.available() > 0) {
                    int len = Math.min(bytesToRead, blockInputStream.available());
                    int count = blockInputStream.read(bytes, offset, len);
                    read += count;
                    offset += count;
                    bytesToRead -= count;
                    n += count;
                } else {
                    readBlock();
                }
            }
        }
        return n;
    }

    @Override
    public int available () throws IOException {
        long remaining = totalLength - read;
        if(remaining < 1){
            return -1;
        }
        return inputStream.available();
    }

    @Override
    public long skip(long n) throws IOException {
        return inputStream.skip(n);
    }

    @Override
    public void close() throws IOException {
        inputStream.close();
    }

    @Override
    public synchronized void mark(int readlimit) {
        inputStream.mark(readlimit);
    }

    @Override
    public synchronized void reset() throws IOException {
        inputStream.reset();
    }

    @Override
    public boolean markSupported() {
        return inputStream.markSupported();
    }
}

This is my final working implementation using this two classes

private RangeData getRangeData(RangeInfo r) throws IOException, GeneralSecurityException, CryptoException {

    // used for encrypted files
    long blockStart = r.getStart();
    long blockEnd = r.getEnd();
    long blockLength = blockEnd - blockStart + 1;

    // encrypted files, it must be access in blocks of 16 bytes
    if(datafile.isEncrypted()){
        blockStart -= blockStart % 16;
        blockEnd = blockEnd | 15; // nearest multiple of 16 for length n = ((n−1)|15)+1
        blockLength = blockEnd - blockStart + 1;
    }

    try ( final FileChannel channel = FileChannel.open(services.getPhysicalFile(datafile).toPath()) )
    {
        MappedByteBuffer mappedByteBuffer = channel.map(FileChannel.MapMode.READ_ONLY, blockStart, blockLength);
        InputStream inputStream = new ByteBufferInputStream(mappedByteBuffer);
        if(datafile.isEncrypted()) {
            String encryptionKeyRef = (String) settingsManager.getSetting(AppSetting.DEFAULT_ENCRYPTION_KEY);
            inputStream = new FileEncryptionUtil().getCipherInputStream(inputStream,
                    EncodeUtil.decodeSeedValue(encryptionKeyRef), AESCipher.DECRYPT_MODE);
            long skipStart = r.getStart() - blockStart;
            inputStream = new BlockInputStream(inputStream, skipStart, r.getLength()); // this will trim the data to n bytes at last
        }
        return new RangeData(r, inputStream);
    }
}