Why is CDOSYS sending me a type 3 message with a 56 byte response?

Question

I have some really old code that I'd like to get working with the CDO.Message object. Many years ago, I think this code did work (Windows 98?), but then it didn't work starting with Windows XP (now I'm on Windows 10).

My code first checks sResponse2.cLength, and if it's 24, then it performs the NTLM calculation. Failing that, if sResponse1.cLength is also 24, then it performs the LM calculation instead. That was all I ever had implemented.

Today, with the CDO.Message testing, it took the LM code path, and the hash that my code calculated did not match what CDO.Message provided. That's another problem.

What I'm more interested in right now is why sResponse2.cLength is 56. What does a value of 56 indicate? Given that value and the flags, should I be testing the type 3 message response using a different algorithm?

When I send the type 2 message to the client, I'm only specifying these two flags:

#define F_NEGOTIATE_OEM             0x00000002
#define F_NEGOTIATE_NTLM            0x00000200

As the code shows, the authentication protocol negotiate to *[NTLMSSP](https://learn.microsoft.com/en-us/windows/win32/com/ntlmssp)*. — YangXiaoPo-MSFT, Mar 29 '22 at 09:23
@YangXiaoPo-MSFT Right, so if the client uses NTLMSSP, and the "sResponse2" field has a length of 56 bytes, then what does that mean? Is it NTLMv2? Is it LMv2? How do I determine which algorithm is being used? — David, Apr 01 '22 at 18:27
@YangXiaoPo-MSFT That's what I was wondering too. I have implemented the algorithm as specified at http://davenport.sourceforge.net/ntlm.html, but my understanding was that the hash would still be 24 bytes. If the client sends 56 bytes in the "sResponse2" field, then where is the 24-byte NTLMv2 hash? — David, Apr 07 '22 at 16:11
@YangXiaoPo-MSFT I think I know how to crack that data now, and it won't even use the "NTLM2 Session Response" code that I had previously written (not sure when that would be used). I will post my code later today. — David, Apr 07 '22 at 21:54

score 0 · Accepted Answer · answered Apr 08 '22 at 04:30

The response was greater than 24 bytes because it was an NTLMv2 response. The 56 bytes include the 16-byte HMAC-MD5 hash, a blob signature, two "reserved" fields, an 8-byte time stamp, an 8-byte client nonce, and variable length target data (12 bytes of overhead in this case). 56 bytes may be the minimum possible size with an empty target server/domain string.

Had it been 24 bytes, it could have instead been an NTLMv2 Session Response, which is different.

Details can be found here: http://davenport.sourceforge.net/ntlm.html

I've created the following code to parse the NTLMv2 response:

struct TYPE3_BLOCK
{
    const BYTE* pcbHash;        // 16 bytes

    const BYTE* pcbBlockPtr;    // Points to the block after the hash
    DWORD cbBlockPtr;

    DWORD dwSignature;
    FILETIME ftStamp;
    const BYTE* pcbNonce;       // 8 bytes
    DWORD dwReserved1;
    DWORD dwReserved2;
    const BYTE* pcbTarget;      // Remainder, less dwReserved2
    DWORD cbTarget;
};

template <typename T>
BOOL TConsumePtr (const BYTE*& pcbData, DWORD& cbData, T** pptPtr, DWORD cbPtr)
{
    if(cbData >= cbPtr)
    {
        *pptPtr = reinterpret_cast<T*>(pcbData);
        pcbData += cbPtr;
        cbData -= cbPtr;
        return TRUE;
    }

    return FALSE;
}

template <typename T>
BOOL TConsumeData (const BYTE*& pcbData, DWORD& cbData, T* ptPtr)
{
    if(cbData >= sizeof(T))
    {
        CopyMemory(ptPtr, pcbData, sizeof(T));
        pcbData += sizeof(T);
        cbData -= sizeof(T);
        return TRUE;
    }

    return FALSE;
}

BOOL CrackType3Response (const BYTE* pcbResponse, DWORD cbResponse, __out TYPE3_BLOCK* pBlock)
{
    if(TConsumePtr(pcbResponse, cbResponse, &pBlock->pcbHash, 16))
    {
        pBlock->pcbBlockPtr = pcbResponse;
        pBlock->cbBlockPtr = cbResponse;

        if(TConsumeData(pcbResponse, cbResponse, &pBlock->dwSignature) &&
            TConsumeData(pcbResponse, cbResponse, &pBlock->dwReserved1) &&
            TConsumeData(pcbResponse, cbResponse, &pBlock->ftStamp) &&
            TConsumePtr(pcbResponse, cbResponse, &pBlock->pcbNonce, 8))
        {
            pBlock->cbTarget = cbResponse - sizeof(pBlock->dwReserved2);

            return TConsumePtr(pcbResponse, cbResponse, &pBlock->pcbTarget, pBlock->cbTarget) &&
                TConsumeData(pcbResponse, cbResponse, &pBlock->dwReserved2);
        }
    }

    return FALSE;
}

Why is CDOSYS sending me a type 3 message with a 56 byte response?

1 Answers1