AES 128 bit CTR partial file decryption with PHP

Question

This is not a duplicate post because I have looked everywhere and I can't find an answer to this. Its about partial decryption. Not full.

I have a good knowledge of PHP but little knowledge about cryptography. I know the key and the iv of the crypted file. The file is decrypting fine as whole but the real issue arises when I try to decrypt the partial file from the middle.

It decrypts fine when I try to decrypt the first 128kb of the file or 256kb or any length from the beginning of the file. but when I start from the middle, it does not decrypt but gives gibberish output.

I will post here an example of first 100 bytes of the file.

The encryption is AES 128 bit CTR mode.

I have used both mdecrypt_generic and mcrypt_decrypt functions of PHP with no success.

code used:

$chunk2 = mcrypt_decrypt(MCRYPT_RIJNDAEL_128, base64_decode($mega_key), $data, 'ctr', base64_decode($mega_iv));
echo base64_encode($data) .'<br />'. base64_encode($chunk2);

Result:

9PX2fU83NF3hLc+HFdyHkqfxC4bHWKUQwQHJkNVnYbKCIQrhlHvTKtz8T3Bb0TgBkyBoGHnDCzZs3bu54KLQ8Bv0lzrTVJbzJY5msBfcy7Zi2Z/fLoMm+nvqdGPTNR0uwv45xJ8=
MTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTExMTE=

As you can see. after decryption, the result is the first 100 bytes of the file containing the digit 1 in series. The file is encrypted by Mega.co.nz using JavaScript. According to the documentation at Mega, the file has been encrypted in chunks for partial encryption/decryption.

File encryption

MEGA uses client-side encryption/decryption to end-to-end-protect file transfers and storage. Data received from clients is stored and transmitted verbatim; servers neither decrypt, nor re-encrypt, nor verify the encryption of incoming user files. All cryptographic processing is under the control of the end user. To allow for integrity-checked partial reads, a file is treated as a series of chunks. To simplify server-side processing, partial uploads can only start and end on a chunk boundary. Furthermore, partial downloads can only be integrity-checked if they fulfil the same criterion. Chunk boundaries are located at the following positions: 0 / 128K / 384K / 768K / 1280K / 1920K / 2688K / 3584K / 4608K /. (every 1024 KB) / EOF

I am calculating the chunk boundaries of the file with this function:

public function get_chunks($size)
{
    $chunks = array();
    $p = $pp = 0;

    for ($i = 1; $i <= 8 && $p < $size - $i * 0x20000; $i++) {
        $chunks[$p] = $i * 0x20000;
        $pp = $p;
        $p += $chunks[$p];
    }

    while ($p < $size) {
        $chunks[$p] = 0x100000;
        $pp = $p;
        $p += $chunks[$p];
    }

    $chunks[$pp] = ($size - $pp);
    if (!$chunks[$pp])
    {
        unset($chunks[$pp]);
    }

    return $chunks;
}

I have tried decrypting the 2nd chunk individually, it fails. I cannot post a 128kb chunk here for obvious reasons, but I will show you chunk starting from 2nd byte to the 100th byte. This is the result with the same code:

code used:

$chunk2 = mcrypt_decrypt(MCRYPT_RIJNDAEL_128, base64_decode($mega_key), $data, 'ctr', base64_decode($mega_iv));
echo base64_encode($data) .'<br />'. base64_encode($chunk2);

Result:

9fZ9Tzc0XeEtz4cV3IeSp/ELhsdYpRDBAcmQ1WdhsoIhCuGUe9Mq3PxPcFvROAGTIGgYecMLNmzdu7ngotDwG/SXOtNUlvMljmawF9zLtmLZn98ugyb6e+p0Y9M1HS7C/jnEnw==
MDK6A0kyWI3903mj+GokBGfLvHCuzITg8flodIM34gGSGtpE3pnIxxGCDhq72AijgnlBUIv5DGuAVzNoc0MR2t5SnNi281TnmtnnlvomTOWKd3HAnJTtsKCvJoHXGQLdDfbMag==

The results are the same with mcrypt_module_open('rijndael-128', '', 'ctr', '');

I need to partially decrypt a file because I am trying to code an open source download manager for Mega which supports parallel connections/resume support.

I need to decrypt the file on the fly to allow file streaming, so decrypting after its downloaded is out of question.

At mega's website, their own download interface uses multiple connections and downloads the file in chunks and there is another web service that allows downloading from Mega with multiple connections and resume support.

I need to decrypt the partial file to support HTTP Range header requests from the browser/download managers. If the range request falls with in the 2nd or 3rd block I need to be able to decrypt that block and send it to the client without decrypting the file from the beginning.

Is it even possible? It should be because some website has already done it.

It's not clear from MEGA doc provided if each chunk is encrypted separately or all chunks are considered as a single data stream. Can you provide any links to this kind of details? — Oleg Gryb, Jul 18 '14 at 00:36
I agree that the documentation is very vague. here is the link: https://mega.co.nz/#doc — user3847106, Jul 18 '14 at 00:51
You are showing us your data as a lot of characters. Don't do that. Show them in Base-64 or in Hex (=Base-16). There are a great many things that can go wrong when trying to express byte data as raw character strings. One of the things that can go wrong is that characters 'go missing', for example: `\r\n` -> `\n`. Be very clear that you are handling bytes, not characters, and translate carefully between bytes and Base-64. — rossum, Jul 18 '14 at 12:46
@rossum Thank you for pointing that out, you are right. But the question is still the same, can the file be decrypted partially? From what I understood from MEGA's docs, the file has been encrypted chunk by chunk and there is a chunk boundary also given but the decryption is not working out as expected. Is there any other way of achieving this? I have tried decrypting it in python's library as well and some other custom made AES decryption algorithm in PHP. — user3847106, Jul 18 '14 at 21:53
A file encrypted with CTR can be decrypted either block by block or chunk by chunk. Blocks within a chunk should run sequentially on the counter. You need to contact the encryption end to determine how the encryption is set up at the start of each chunk: where does the counter (re)start? what nonce is used? is a new key used? All start-of-chunk conditions need to be reproduced exactly, byte-for-byte, if the decryption is to succeed. — rossum, Jul 18 '14 at 22:08
It says: file data is encrypted in ctr mode with a 64 bit random counter start value. Further it says about the blocks: `For each AES block d: h := AES(k,h XOR d) A chunk is encrypted using standard counter mode: For each AES block d at block position p: d' := d XOR AES(k,(n << 64)+p)` the file is encrypted chunk by chunk (starting and ending at the chunk boundaries given in the original post). To generate the IV, we generate a random 128 bit AES key for the file, and the upper 64 bits of the counter start value (initialization vector). — user3847106, Jul 18 '14 at 22:34
With that key and IV, the whole file can be encrypted at once by using the mcrypt function and uploaded OR encrypted chunk by chunk using the get_chunks function I posted above to compute the chunk boundaries. The key does not change at all. The same key is used for all the chunks. I have already mentioned where the counter starts and restarts: `Chunk boundaries are located at the following positions: 0 / 128K / 384K / 768K / 1280K / 1920K / 2688K / 3584K / 4608K /. (every 1024 KB) / EOF` If thats what you meant by counter start position. As for the counter start value, it doesn't change. — user3847106, Jul 18 '14 at 22:37

score 1 · Answer 1 · answered Aug 22 '14 at 22:25

Okay, I found the solution about decrypting partial content on the fly. I had this problem when I tried to decrypt partial data from Range ( in order to create a web proxy ). I discovered that if I download whole the file (from position 0 so), there is no problem. Example :

while($_total_dled != $content_length) {
        $raw = fgets($socket, 1024);
        $data = mdecrypt_generic($this->cipher, $raw);
        echo $data;
}

BUT, if I start reading the file from pos x ( like if I received : Range x-) , mdecrypt_generic will failed, because it wasn't be executed (x-1) times before. So I try this :

for($i=0;$i<$range["start"];$i++) {
     mdecrypt_generic($this->cipher, "0");
}

Just need to decrypt one character (x-1) times. and then, the part with the while loop. And it works. May be we can link this iteration with the related counter.

Hope this will help people.

score 0 · Answer 2 · answered Jul 20 '14 at 11:10

0

Seems to me that you are doing a lot of calculations on the chunk boundaries (which are given as a table + simple calculation over 4608Ki) and no calculation to get to a new IV.

The IV is just the random nonce plus a block counter. So to calculate a new "IV" for a specific chunk you would have to do:

create the initial counter by shifting the nonce in the higher (left) bytes
get the chunk lower boundary
divide the lower boundary by the block size
add or XOR (the same for the first 2^64 blocks) the resulting number to the initial counter
initialize your cipher with it

Then you should be able to decrypt.

answered Jul 20 '14 at 11:10

Maarten Bodewes

90,524
13
150
263

okay this answer was helpful, but does not solve the problem unfortunately. Like I said I know little to no knowledge of cryptography. But I am not creating the IV myself, its already provided to me by MEGA encoded in the file key. I am doing this to get the IV. what I find interesting is, that the IV value is always an array consisting of 0. Array ( [0] => 0 [1] => 0 ) packed into big endian 32 bit long string. This array is obtained by slicing the file key array (after unpacking it with the same 32 bit long big endian format. – user3847106 Jul 22 '14 at 04:09
in php it is done by using the unpack() function by passing N* as the format. The file key is sliced from the 4th offset and the length of the slice is 2. but it yields and empty array because there are only 4 elements of that array. hence we get an array of length two with no elements. then that IV array is converted into the same 32bit string and used in decryption. I understood that you want me to create a new IV for the middle block. But I'm blank. I could not understand what you meant by: `create the initial counter by shifting the nonce in the higher (left) bytes` – user3847106 Jul 22 '14 at 04:17
Can you please explain what I have to do in layman's terms? That would be very generous of you. – user3847106 Jul 22 '14 at 04:17

AES 128 bit CTR partial file decryption with PHP

2 Answers2

Linked