Edit (2018-09-26): I've discovered that disabling 3DES on 2012R2 does NOT break RDP but it DOES break on 2008 R2. The options supported appear to be different between the kernels.
I'll share my answer from a TechNet thread but first BLUF:
Serverfault conclusion: Most likely you have some other difference in between the systems. You're connecting between different OS versions, one system has FIPS enabled and the other does not, or you have different cipher restrictions in place under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers. I would certainly enable the SCHANNEL logging on the system that does work to determine which cipher is in use. Would love to hear back if you somehow got RDP to work with an alternate cipher.
Copy of post:
We got it to work!
Apparently 2008 and 2012 have syntax issues and the 2008/7 requires a trailing /168. 2012/8.1/10 does not.
the key on 2008 looks like this: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\Triple DES 168/168
And the key on 2012 looks like this: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\Triple DES 168
I can confirm that use of "Triple DES 168/168" DOES NOT disable 3DES on the system. You can prove this to yourself with a protocol scanner (like Nessus) or by enabling SCHANNEL logging:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL] "EventLogging"=dword:00000007
You will then have events in the SYSTEM log for example;
An SSL client handshake completed successfully. The negotiated cryptographic parameters are as follows.
Protocol: TLS 1.0 CipherSuite: 0x2f Exchange strength: 1024
For me the result is 0xa which Google reveals as TLS_RSA_WITH_3DES_EDE_CBC_SHA.
When I use "Triple DES 168" (without the /168), the System event ID 36880 does not appear and the RDP session is blocked.
Per the article: System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing
Remote Desktop Services (RDS)
For encrypting Remote Desktop Services network communication, this policy setting supports only the Triple DES encryption algorithm.
Per the article: "System cryptography: Use FIPS compliant algorithms for encryption, hashing, and signing" security setting effects in Windows XP and in later versions of Windows
This setting also affects Terminal Services in Windows Server 2003 and in later versions of Windows. The effect depends on whether TLS is being used for server authentication.
If TLS is being used for server authentication, this setting causes only TLS 1.0 to be used.
By default, if TLS is not being used, and this setting is not enabled on the client or on the server, the Remote Desktop Protocol (RDP) channel between the server and the client is encrypted by using the RC4 algorithm with a 128-bit key length. After you enable this setting on a Windows Server 2003-based computer, the following is true:
The RDP channel is encrypted by using the 3DES algorithm in Cipher Block Chaining (CBC) mode with a 168-bit key length.
The SHA-1 algorithm is used to create message digests.
Clients must use the RDP 5.2 client program or a later version to connect.
So both of these support the idea that RDP can only utilize 3DES. However, this article suggests a larger range of ciphers is available: FIPS 140 Validation
The set of cryptographic algorithms that a Remote Desktop Protocol (RDP) server will use is scoped to:
- CALG_RSA_KEYX - RSA public key exchange algorithm
- CALG_3DES - Triple DES encryption algorithm
- CALG_AES_128 - 128 bit AES
- CALG_AES_256 - 256 bit AES
- CALG_SHA1 - SHA hashing algorithm
- CALG_SHA_256 - 256 bit SHA hashing algorithm
- CALG_SHA_384 - 384 bit SHA hashing algorithm
- CALG_SHA_512 - 512 bit SHA hashing algorithm
Ultimately it's not clear whether RDP can support non-3DES protocols when FIPS mode is enabled but evidence would suggest it doesn't.
I see no evidence that Server 2012 R2 would function differently from Server 2008 R2, however it does seem that Server 2008 R2 was based around FIPS 140-1 compliance and Server 2012 R2 follows FIPS 140-2 so it's entirely possible that Server 2012 R2 supports additional protocols. You'll note the additional protocols in the FIPS 140 Validation link.
In conclusion: I don't think Server 2008 R2 can support RDP in FIPS mode with 3DES disabled. My recommendation is to ascertain whether your system meets the conditions for a SWEET32 attack (more than 768GB sent in a single session) and whether disabling 3DES is worth removing RDP capability. Other utilities exist to manage servers beyond RDP especially in a world where virtualization is highly commonplace.
