3

I'm currently working on a Java TLS server. I'm trying to get the following CipherSuite to work : TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

When I test it using openssl s_client I get the following error after the ServerKeyExchange message :

140735242416208:error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type:t1_lib.c:1130:

Here is the TLS message as seen in Wireshark The TLS message as seen in Wireshark

The Handshake fails on a decode_error fatal error.

So I guess the client doesn't like the signature algorithm chosen.

But I am only using the default SignatureAndHashAlgorithm for now as per RFC 5246 Section-7.4.1.4.1

If the negotiated key exchange algorithm is one of (RSA, DHE_RSA, DH_RSA, RSA_PSK, ECDH_RSA, ECDHE_RSA), behave as if client had sent the value {sha1,rsa}.

(I'm still checking if the client do offer theses default values though)

Since I'm doing ECDHE_RSA I believe I should hash and sign the serverECDHparams as per RFC 4492 Section 5.4 (First post here so only 2 links sorry :) )

ServerKeyExchange.signed_params.sha_hash
        SHA(ClientHello.random + ServerHello.random +
                                          ServerKeyExchange.params);
struct {
    select (KeyExchangeAlgorithm) {
        case ec_diffie_hellman:
            ServerECDHParams params;
            Signature signed_params;
    };
} ServerKeyExchange;

And I should do this as per RFC 2246 Section 7.4.3

select (SignatureAlgorithm) {   
    case rsa:
        digitally-signed struct {
            opaque md5_hash[16];
            opaque sha_hash[20];
        };
} Signature;


md5_hash
MD5(ClientHello.random + ServerHello.random + ServerParams);

sha_hash
SHA(ClientHello.random + ServerHello.random + ServerParams);

My Java code regarding signing the serverParams :

private byte[] getSignedParams(ChannelBuffer params)
        throws NoSuchAlgorithmException, DigestException, 
        SignatureException, InvalidKeyException {
    byte[] signedParams = null;
    ChannelBuffer signAlg = ChannelBuffers.buffer(2);
    MessageDigest md5 = MessageDigest.getInstance("MD5");
    MessageDigest sha = MessageDigest.getInstance("SHA-1");
    switch (session.cipherSuite.sign) {
        case rsa:
            signAlg.writeByte(2); // 2 for SHA1
            sha.update(clientRandom);
            sha.update(serverRandom);
            sha.update(params.toByteBuffer());
            md5.update(clientRandom);
            md5.update(serverRandom);
            md5.update(params.toByteBuffer());
            signedParams = concat(md5.digest(), sha.digest());
        break;
    }
    signAlg.writeByte(session.cipherSuite.sign.value); // for RSA he byte is one
    ChannelBuffer signLength = ChannelBuffers.buffer(2);
    signLength.writeShort(signedParams.length);
    return concat(signAlg.array(),concat(signLength.array(),signedParams));
}

So my question is basically : Am I wrong about all this ? and if so, what am I doing wrong ?

Thank you for your time ! :)

Community
  • 1
  • 1
Alpha4
  • 41
  • 1
  • 9
  • I would suggest posting this question on the stackoverflow forum for your programming related doubts. You may get answers quickly there – Limit Jul 07 '16 at 09:55
  • Thank you, well I thought this was purely a problem of me not understanding, the RFCs. I've since found an (the?) answer and I'll post it there for people to find later today :) – Alpha4 Jul 07 '16 at 12:45
  • The text you quote from 5246 7.1.4.1 is headed "If the client does not send the signature_algorithms extension". openssl s_client **does** send the extension; look at the ClientHello in your trace -- but it includes 0201 SHA1+RSA and so shouldn't reject this signature for that reason. However, your signature is wrong, as you found; the MD5+SHA1+RSA signature is used in TLS 1.0 (2246) and 1.1 (4346) but 1.2 changes to standard PKCS1-v1_5 with a single, server-chosen, hash; see 5246 4.7 and the second item in the list of differences in 1.2. – dave_thompson_085 Jul 09 '16 at 02:21
  • Thank you @dave_thompson_085 ! I somehow missed that part of the RFC. I will take a closer look at PKCS1-v1_5 then :) – Alpha4 Jul 11 '16 at 07:53
  • Hey @dave_thompson_085, I posted [another question](http://stackoverflow.com/questions/38436853/java-tls-1-2-server-aes-gcm-decryption) regarding TLS1.2 & GCM, you seem to be knowledgeable on the subject. Could you help me once again ? :) – Alpha4 Jul 19 '16 at 13:37

1 Answers1

0

It's me again, I seem to have fixed my particular problem 2 things I noted :

  1. Regarding my Java code, the MessageDigest class only does hashing no signing so I now use the Signature class instead.
  2. It seems I only need to sign using SHA1 in TLS1.2 I don't need to do MD5 at all.

The second item is what I should have found in the RFC but didn't (maybe it is written somewhere, I don't know) I think this could be useful for people even if they're not doing Java ;)

How my code looks now : 

private byte[] getSignedParams(ChannelBuffer params)
        throws NoSuchAlgorithmException, DigestException, 
        SignatureException, InvalidKeyException {
    byte[] signedParams = null;
    Signature signature = Signature.getInstance(selectedSignAndHash.toString());
    ChannelBuffer signAlg = ChannelBuffers.buffer(2);
    signAlg.writeByte(selectedSignAndHash.hash.value);
    signature.initSign(privateKey);

    signature.update(clientRandom);
    signature.update(serverRandom);
    signature.update(params.toByteBuffer());

    signedParams = signature.sign();

    signAlg.writeByte(session.cipherSuite.sign.value);
    ChannelBuffer signLength = ChannelBuffers.buffer(2);
    signLength.writeShort(signedParams.length);
    return concat(signAlg.array(), concat(signLength.array(), signedParams));
}

The code is different because in between I added a function to choose the SignatureAndHashAlgorithm to use from the list the client provides. But you could modify this to only respond using SHA1withRSA as this seems to be the default HashAndSignatureAlgorithm.

Alpha4
  • 41
  • 1
  • 9