Check for errors in TLS implementation

Question

I'm developing a small TLS client, which is used together with SMTP. The handshake is working well until my client sends the encrypted finished message. So Client Hello, Server Hello, Certificate, Server Hello Done, Client Key Exchange and Change Cipher Spec are working. As chiper suite I'm using TLS_RSA_WITH_AES_256_CBC_SHA256.

My problem is, that I receive the "Bad Record MAC" alert from the server after sending the finished message. But I have no idea where I can start to search for the error. I've double checked all my functions and readed the RFC twice.

In my opionion one of the following points can cause the "Bad Record MAC" alert:

The master-secret is wrong.
The client_write_MAC_key is wrong.
The client_write_encryption_key is wrong.
The P_hash function is wrong.
The PRF function is wrong.
The MAC function is wrong.
The AES encryption doesn't work correctly.
The hash of the Finished Message is wrong.

Has anyone an idea what I can do to find the issue. Are there any tools to check if the master-secret, MAC and key calculations are correct? Or is it possible to decrypt the content with Wireshark? Note that I'm not having the private key of the server.

As far as I remember there are counters on server and client for every TLS session. They are added to every message and verified once the message is received on another side. They are used to prevent replay attacks. Please verify that those counters are properly encrypted and MACed. — Sparrow_ua, Sep 12 '16 at 18:23
"...Note that I'm not having the private key of the server..." - then simply setup your own server where you have full control over the certificates, ciphers, protocol etc. It's a bad idea to test against some live system where you don't have any control. It only makes debugging much harder and in the worst case it might be considered abuse. — Steffen Ullrich, Sep 12 '16 at 18:36
@Steffen Ullrich I will think about, setting up a test server with openssl. However does anyone know such tools do check the master-secret and key calculations? — Benjamin J., Sep 12 '16 at 19:50

score 0 · Accepted Answer · answered Sep 13 '16 at 19:20

I've setted up an local test server with OpenSSL as Steffen Ullrich suggested. I've done a detailed analytics of the "debug output" which was provided by Wireshark.

I was able to solve a problem with my padding. But I'm still getting a Bad Record MAC alert. Like the debug output says, the MAC is incorrect (message ssl_decrypt_record: mac failed).

Some more information about the connection and the debug output:

The pre-master-secret, master-secret and all the keys (client write MAC key, server write MAC key, client write key, server write key, client write IV, server write IV) are correct. I've compared these from my software with them of the debug output. So everything until the Encrypted Finished Message is correct.
The server / Wireshark can decrypt the Encrypted Finished Message successfuly.
The server / Wireshark detects the padding correctly.
The server / Wireshark "skips" the IV and only shows the Finished Message and the MAC in the line Plaintext[64]:

In my opinion only two things can cause this error:

The MAC is calculated wrong.
The hash (verify_data) from the handshake messages are wrong.

My Encrypted Finished Message has a full length of 80 bytes and the following structure:

struct
{
    // TLS record
    ContentType type;
    ProtocolVersion version;
    uint16 length;
    // TLS handshake and content (encrypted)
    uint8 IV[16];
    struct
    {
        HandshakeType msg_type;
        uint24 length;
        uint8 verify_data[12];
    } content;
    uint8 MAC[32];
    uint8 padding[15];
    uint8 paddingLength;
} FinishedMessage;

The content looks for example like the following:

     00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
0000 16 03 03 00 50 XX XX XX XX XX XX XX XX XX XX XX
0010 XX XX XX XX XX 14 00 00 0C YY YY YY YY YY YY YY
0020 YY YY YY YY YY ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ
0030 ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ ZZ
0040 ZZ ZZ ZZ ZZ ZZ 0F 0F 0F 0F 0F 0F 0F 0F 0F 0F 0F
0050 0F 0F 0F 0F 0F

Where XX represents the randomly generated IV and YY the verify_data and ZZ the MAC of the message like described in the RFC.

The verify_data contains the first 12 bytes of the SHA-256 hash. The hash is computed by the messages Client Hello, Server Hello, Certificate (from Server), Server Hello Done and Client Key Exchange. For hashing the full messages without the record header (first 5 bytes) are used.

The MAC is computed by the client write MAC key and the message. As sequence number the number 0 is used. The fragment which is used for calculating the MAC only contains the msg_type, length and verify_data (see structure content above).

Has anyone an idea how I can find what is wrong with my MAC?

I've solved the error. My problem was that for calculating the MAC I've used the complette message (including the IV). So the complete MAC was wrong. I've found the problem by reading my post above again and compared it with my software. Sorry for my confusion. — Benjamin J., Sep 14 '16 at 17:40
Could this be related https://stackoverflow.com/questions/73268472/bad-record-mac-on-tls-v1-3-when-using-eary-data ? I appreciate your comments. — xpepermint, Aug 07 '22 at 15:29

Check for errors in TLS implementation

1 Answers1