Decrypting an mcrypt file in .Net (C#)

Question

I've been a C# hobbyist for some time now and would consider having intermediate development skills, but little to no encryption knowledge. As part of a side project I need to decrypt files that were encrypted using MCrypt. It doesn't seem any special arguments were passed into the command. For example, this is quite common (key & filename changed) and the keys are of varying length, anywhere from 14-18 characters.
mcrypt -a rijndael-256 fileToEncrypt.tar.gz -k 0123456789abcdef1

So far, I have taken two approaches to this task. The first is to use mcrypt.exe and start the process using Process. However, I feel that makes the code (and program flow) very clunky. The second is to try to directly decrypt the file from my within program and have zero external program dependencies; I'd like to go this route.

I'm a bit confused with the MCrypt format. I've reviewed the FORMAT doc in the source code (here to view online) and I believe I have the beginning portion of the header taken care of properly. I cannot, however, seem to decrypt the encrypted data within the file.

1) How large is the IV and how do I pass it into my decryptor?
2) How large is the checksum at the end of the file and do I need it?
3) Are the above static in length?
4) What is keymode (mcrypt-sha1) and how is it used?
5) I notice that when properly decrypting (using mcrypt.exe) that there is a 140 byte difference between the encrypted and decrypted file. What makes up these 140 bytes?

Code and the beginning of the encrypted file below; no doubt my code is wrong starting with the comment "Get the data" Any pointers in the right direction would be greatly appreciated.

/// <summary>
/// Decrypt an mcrypt file using rijndael-256
/// </summary>
/// <param name="inputFile">File to decrypt</param>
/// <param name="encryptionKey">Password</param>
/// <param name="purge"></param>
public static bool Decrypt (string inputFile, string encryptionKey)
{
    var rv = false;
    if (File.Exists(inputFile) == true)
    {
        using (FileStream stream = new FileStream(inputFile, FileMode.Open))
        {
            var buffer = new byte[1024];

            // MCrypt header
            stream.Read(buffer, 0, 3);

            if (buffer[0] == 0x00 && buffer[1] == 0x6D && buffer[2] == 0x03)
            {
                // Flag
                // Bit 7 - Salt Used
                // Bit 8 - IV not used
                var flag = (byte)stream.ReadByte();

                byte[] saltVal = null;
                var saltUsed = Utils.GetBit(flag, 6);
                byte[] ivVal = new byte[16];
                var ivUsed = (Utils.GetBit(flag, 7) == false);

                var algorithmName = Utils.GetNullTerminatedString(stream);

                stream.Read(buffer, 0, 2);
                var keyLen = (buffer[1] << 8) + buffer[0];

                var algorithModeName = Utils.GetNullTerminatedString(stream);

                var keygenName = Utils.GetNullTerminatedString(stream);

                if (saltUsed)
                {
                    var saltFlag = (byte)stream.ReadByte();
                    if (Utils.GetBit(saltFlag, 0))
                    {
                        // After clearing the first bit the salt flag is now the length
                        Utils.ClearBit (ref saltFlag, 0);
                        saltVal = new byte[saltFlag];
                        stream.Read(saltVal, 0, saltFlag);
                    }
                }

                var algorithmModeName = Utils.GetNullTerminatedString(stream);

                if (ivUsed)
                {
                    stream.Read(ivVal, 0, ivVal.Length);
                }

                // Get the data - how much to get???
                buffer = new byte[stream.Length - stream.Position + 1];
                var bytesRead = stream.Read(buffer, 0, buffer.Length);

                using (MemoryStream ms = new MemoryStream())
                {
                    using (RijndaelManaged rijndael = new RijndaelManaged())
                    {
                        rijndael.KeySize = 256;
                        rijndael.BlockSize = 128;

                        var key = new Rfc2898DeriveBytes(System.Text.Encoding.ASCII.GetBytes(encryptionKey), saltVal, 1000);
                        rijndael.Key = key.GetBytes(rijndael.KeySize / 8);
                        //AES.Key = System.Text.Encoding.ASCII.GetBytes(encryptionKey);
                        //AES.IV = key.GetBytes(AES.BlockSize / 8);
                        rijndael.IV = ivVal;

                        rijndael.Mode = CipherMode.CBC;
                        rijndael.Padding = PaddingMode.None;

                        using (var cs = new CryptoStream(ms, rijndael.CreateDecryptor(), CryptoStreamMode.Write))
                        {
                            cs.Write(buffer, 0, buffer.Length);
                            cs.Close();

                            using (FileStream fs = new FileStream(inputFile + Consts.FILE_EXT, FileMode.Create))
                            {
                                byte[] decryptedBytes = ms.ToArray();
                                fs.Write(decryptedBytes, 0, decryptedBytes.Length);
                                fs.Close();
                                rv = true;
                            }
                        }
                    }
                }
            }
        }
    }

    return rv;
}

Edit
I receive the following when turning on its verbose mode and without specifying rijndael-256. When I do specify the algorithm it does reflect that in the verbose output; both decrypt the file properly. The plot thickens...

Algorithm: rijndael-128
Keysize: 32
Mode: cbc
Keyword mode: mcrypt-sha1
File format: mcrypt

Also, "passwords" used to encrypt in various parts of the software range from 12 to 28 characters.

Answer 1

MCrypt file format

Observations were made using mcrypt-2.6.7-win32, encrypting the following file with the command mcrpyt.exe --no-openpgp -V test_in.txt

test_in.txt unencrypted is 25 bytes in length, and the above command encrypts as follows, resulting in the file test_out.txt.nc which is 125 bytes in length.

+-------------+----------------------+----------------+---------------------------------------------+
| File Offset | Field Length (bytes) | Field Content  | Description                                 |
+-------------+----------------------+----------------+---------------------------------------------+
| 0           | 1                    | 0x0            | Zero byte                                   |
+-------------+----------------------+----------------+---------------------------------------------+
| 1           | 1                    | 0x6d           | m                                           |
+-------------+----------------------+----------------+---------------------------------------------+
| 2           | 1                    | 0x3            | Version                                     |
+-------------+----------------------+----------------+---------------------------------------------+
| 3           | 1                    | 0x40           | Flags - bit 7 set = salt, bit 8 set = no IV |
+-------------+----------------------+----------------+---------------------------------------------+
| 4           | 13                   | rijndael-128   | Algorithm name                              |
+-------------+----------------------+----------------+---------------------------------------------+
| 17          | 2                    | 32             | Key Size                                    |
+-------------+----------------------+----------------+---------------------------------------------+
| 19          | 4                    | cbc            | Algorithm mode                              |
+-------------+----------------------+----------------+---------------------------------------------+
| 23          | 12                   | mcrypt-sha1    | Key generator algorithm                     |
+-------------+----------------------+----------------+---------------------------------------------+
| 35          | 1                    | 21             | Salt length + 1                             |
+-------------+----------------------+----------------+---------------------------------------------+
| 36          | 20                   | Salt data      | Salt                                        |
+-------------+----------------------+----------------+---------------------------------------------+
| 56          | 5                    | sha1           | Check sum algorithm                         |
+-------------+----------------------+----------------+---------------------------------------------+
| 61          | 16                   | IV data        | Initialisation vector                       |
+-------------+----------------------+----------------+---------------------------------------------+
| 77          | 48                   | Encrypted data | 25 original data + 20 check sum + 3 padding |
+-------------+----------------------+----------------+---------------------------------------------+
| TOTAL       | 125                  |                |                                             |
+-------------+----------------------+----------------+---------------------------------------------+

Observing the output in different scenarios, the following block/key/IV sizes are used:

+--------------+--------------------+------------+------------------+
| Algorithm    | Block Size (bytes) | IV (bytes) | Key Size (bytes) |
+--------------+--------------------+------------+------------------+
| rijndael-128 | 16                 | 16         | 32               |
+--------------+--------------------+------------+------------------+
| rijndael-256 | 32                 | 32         | 32               |
+--------------+--------------------+------------+------------------+

The check sum is done on the original data before encryption, and appended to the end of the original data. The default check sum algorithm used is SHA-1 which results in a 20 byte hash. So, the original data of 25 bytes becomes 45 bytes. With a block size of 128 bits (16 bytes), that results in 3 bytes of padding to reach the block size of 48 bytes. With a block size of 256 bits (32 bytes), there would be 19 bytes of padding to get to 64 bytes. Zero bytes are used for padding, which is significant during decryption as these are not automatically removed since the size of the original data is not known.

Reading the header

Here is a code sample of reading the header and encrypted data at the tail of the file. Not all helper functions are included for brevity.

public void ReadHeader(Stream stream)
{
    byte[] buffer = new byte[512];
    stream.Read(buffer, 0, 3);
    if (buffer[0] != 0x0) throw new FormatException($"First byte is not 0x0, invalid MCrypt file");
    if ((char)buffer[1] != 'm') throw new FormatException($"Second byte is not null, invalid MCrypt file");
    if (buffer[2] != 0x3) throw new FormatException($"Third byte is not 0x3, invalid MCrypt file");

    byte flags = (byte)stream.ReadByte();
    KeyGeneratorUsesSalt = (flags & (1 << 6)) != 0;
    HasInitialisationVector = (flags & (1 << 7)) != 1;
    AlgorithmName = ReadNullTerminatedString(stream);
    stream.Read(buffer, 0, 2);
    KeySize = BitConverter.ToUInt16(buffer, 0);
    BlockSize = GetBlockSize(AlgorithmName);

    var cipherModeAsString = ReadNullTerminatedString(stream);
    CipherMode cipherMode;
    if (Enum.TryParse<CipherMode>(cipherModeAsString, out cipherMode))
        CipherMode = cipherMode;

    KeyGeneratorName = ReadNullTerminatedString(stream);

    if (KeyGeneratorUsesSalt)
    {
        var saltSize = ((byte)stream.ReadByte()) - 1;
        Salt = new byte[saltSize];
        stream.Read(Salt, 0, saltSize);
    }

    CheckSumAlgorithmName = ReadNullTerminatedString(stream);

    if (HasInitialisationVector)
    {
        InitialisationVector = new byte[BlockSize / 8];
        stream.Read(InitialisationVector, 0, BlockSize / 8);
    }

    int read = 0;
    byte[] remainingData = null;
    using (MemoryStream mem = new MemoryStream())
    {
        while ((read = stream.Read(buffer, 0, buffer.Length)) != 0)
        {
            mem.Write(buffer, 0, read);
            remainingData = mem.ToArray();
        }
    }

    EncryptedData = remainingData;
}

Key Generation

The key generator algorithm is specified in the header and by default in MCrypt format is mcrypt-sha1. Looking into the mcrypt source, that key is generated using the mhash library. It combines the passphrase with the salt to produce a key of the required number of bytes for the algorithm (32 bytes in both the cases I looked at). I translated the function _mhash_gen_key_mcrypt from the mhash library into C# as below - perhaps it's already in the .NET framework somewhere, but if so I couldn't find it.

public byte[] GenerateKeyMcryptSha1(string passPhrase, byte[] salt, int keySize)
{
    byte[] key = new byte[KeySize], digest = null;
    int hashSize = 20;
    byte[] password = Encoding.ASCII.GetBytes(passPhrase);
    int keyBytes = 0;

    while (true)
    {
        byte[] inputData = null;
        using (MemoryStream stream = new MemoryStream())
        {
            if (Salt != null)
                stream.Write(salt, 0, salt.Length);
            stream.Write(password, 0, password.Length);
            if (keyBytes > 0)
                stream.Write(key, 0, keyBytes);
            inputData = stream.ToArray();
        }

        using (var sha1 = new SHA1Managed())
            digest = sha1.ComputeHash(inputData);

        if (keySize > hashSize)
        {
            Buffer.BlockCopy(digest, 0, key, keyBytes, hashSize);
            keySize -= hashSize;
            keyBytes += hashSize;
        }
        else
        {
            Buffer.BlockCopy(digest, 0, key, keyBytes, keySize);
            break;
        }                
    }

    return key;
}

Decryption

We can use standard .NET crypto classes to do most of the decryption, passing in 32-byte key we generated by hashing the passphrase and salt and where we're using the 128-bit or 256-bit flavour based on the algorithm name from the header. We assign the initialisation vector (IV) we read from the header via rijndael.IV = InitialisationVector;.

/// <summary>
/// Decrypt using Rijndael
/// </summary>
/// <param name="key">Key to use for decryption that was generated from passphrase + salt</param>
/// <param name="keySize">Algo key size, e.g. 128 bit, 256 bit</param>
/// <returns>Unencrypted data</returns>
private byte[] DecryptRijndael(byte[] key, int keySize)
{
    using (RijndaelManaged rijndael = GetRijndael(key, keySize))
    {
        rijndael.IV = InitialisationVector;
        using (MemoryStream unencryptedStream = new MemoryStream())
        using (MemoryStream encryptedStream = new MemoryStream(EncryptedData))
        {
            using (var cs = new CryptoStream(encryptedStream, rijndael.CreateDecryptor(), CryptoStreamMode.Read))
                cs.CopyTo(unencryptedStream);

            byte[] unencryptedData = RemovePaddingAndCheckSum(unencryptedStream.ToArray(), GetCheckSumLen());                    
            return unencryptedData;
        }
    }
}

/// <summary>
/// Set algorithm mode/settings
/// </summary>
/// <param name="key">Key to use for decryption that was generated from passphrase + salt</param>
/// <param name="keySize">Algo key size, e.g. 128 bit, 256 bit</param>
/// <returns>Instance ready to decrypt</returns>
private RijndaelManaged GetRijndael(byte[] key, int keySize)
{
    var rijndael = new RijndaelManaged()
    {
        Mode = CipherMode, // e.g. CBC
        KeySize = keySize, // e.g. 256 bits
        Key = key, // e.g. 32-byte sha-1 hash of passphrase + salt
        BlockSize = BlockSize, // e.g. 256 bits
        Padding = PaddingMode.Zeros
    };

    return rijndael;
}

Since the padding style is zero-bytes, these are not removed during decryption as we don't know the size of the original data at that point, so the decrypted data will always be a multiple of the block size no matter the size of the original data. It's also going to have the checksum appended to the end. We could simply remove all zero bytes from the tail of the decrypted block, but we'd risk corrupting the check sum and original data if that really ended on a zero byte.

So instead we could work backwards one byte at a time from the tail and use the check sum to validate when we have the correct original data.

/// <summary>
/// Remove zero padding by starting at the end of the data block assuming
/// no padding, and using the check sum appended to the end of the data to
/// verify the original data, incrementing padding until we match the 
/// check sum or conclude data is corrupt
/// </summary>
/// <param name="data">Decrypted data block, including zero padding and checksum at end</param>
/// <param name="checkSumLen">Length of the checksum appended to the end of the data</param>
/// <returns>Unencrypted original data without padding and without check sum</returns>
private byte[] RemovePaddingAndCheckSum(byte[] data, int checkSumLen)
{
    byte[] checkSum = new byte[checkSumLen];
    int padding = 0;

    while ((data.Length - checkSumLen - padding) > 0)
    {
        int checkSumStart = data.Length - checkSumLen - padding;
        Buffer.BlockCopy(data, checkSumStart, checkSum, 0, checkSumLen);
        int dataLength = data.Length - checkSumLen - padding;
        byte[] dataClean = new byte[dataLength];
        Buffer.BlockCopy(data, 0 , dataClean, 0, dataLength);

        if (VerifyCheckSum(dataClean, checkSum))
            return dataClean;

        padding++;
    }

    throw new InvalidDataException("Unable to decrypt, check sum does not match");
}

The SHA1 20 byte check sum can be validated against the data simply as follows:

private bool VerifySha1Hash(byte[] data, byte[] checkSum)
{
    using (SHA1Managed sha1 = new SHA1Managed())
    {
        var checkSumRedone = sha1.ComputeHash(data);
        return checkSumRedone.SequenceEqual(checkSum);
    }
}

And that's it, with the 128-bit after 3 attempts we should get the right check sum and corresponding original data, which we then return to the caller as the unencrypted original data.