5

I don't understand how does Java picks the most prefered cipher to use on Server Hello.

I have a Tomcat 5 configuration and I set in the SSL connector the ciphers=TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, etc

Server i.e. Java picks TLS_RSA_WITH_AES_128_CBC_SHA on Server Hello as the prefered among the client's supported. But this is not the most secure and it is not the prefered accoding to http://docs.huihoo.com/java/javase/7/technotes/guides/security/SunProviders.html#SunJSSEProvider which lists that TLS_RSA_WITH_AES_256_CBC_SHA has preference.
Then I thought it was the order in the server.xml attribute that made the difference and I put another cipher first (TLS_DHE_RSA_WITH_AES_256_CBC_SHA) which I see in the Client Hello that it is supported. But this was not selected either and TLS_RSA_WITH_AES_128_CBC_SHA was again selected.

So how does JSSE picks the cipher of preference? Is this documented somewhere? I can not figure out what's going on here.

Jim
  • 18,826
  • 34
  • 135
  • 254

3 Answers3

4

It doesn't have to. All that RFC 2246 says is "The server will select a cipher suite". Nowhere does it say it will pick the most secure, or indeed anything about how it will make that selection.

user207421
  • 305,947
  • 44
  • 307
  • 483
  • 1
    Strange. All literature says that the server picks the most secure e.g. http://en.wikipedia.org/wiki/Transport_Layer_Security as a trivial sample. Am I wrong on this? But even if it is left unspecified as you pointed out, it seems that SUN advertises a preference order http://docs.oracle.com/javase/6/docs/technotes/guides/security/SunProviders.html. It says: `The table below shows the ciphersuites supported by SunJSSE in their default preference order ` So it seems that there is a preference order or am I misinterpreting it here? – Jim Apr 24 '12 at 10:35
  • 2
    I mean here http://docs.oracle.com/javase/6/docs/technotes/guides/security/SunProviders.html – Jim Apr 25 '12 at 05:50
  • @Jim The Wikipedia citation doesn't say anything that I can see about selecting the strongest cipher suite, and if it does it is going beyond the RFC. If JSSE is documented to use a specific precedence order for cipher suites, that is how it should behave, but it isn't required by RFC2246. – user207421 Apr 25 '12 at 10:26
  • From wikipedia: `From this list, the server picks the strongest cipher and hash function that it also supports and notifies the client of the decision`. Ok it is against RFC. I can not find out if the default preference order of JSSE can change (I mean if they name it `default` I would expect it to be configurable) – Jim Apr 25 '12 at 10:54
  • 1
    @Jim So Wikipedia is wrong. Wouldn't be the first time. I've fixed it. The RFC does say that the client specifies *his* order of preference, but it's the server's choice. You can change the enabled cipher suites in JSSE with `setEnabledCipherSuites()`, but there's nothing in the Javadoc that says the order you specify becomes the order of preference. – user207421 Apr 25 '12 at 11:01
  • So if this is the case, if someone wants to specify a specific security policy, e.g. `DHE` in preference of `RSA` or larger key sizes ignoring performance, how is he supposed to do that? If JSSE doesn't offer this configurability, what am I supposed to do? Create my own secure sockets? I mean is my expectation wrong or absurd? Shouldn't I be able to define what the security options should be (strong or lenient)? – Jim Apr 25 '12 at 12:51
  • @Jim I found this in the JSSE Reference Guide (http://docs.oracle.com/javase/6/docs/technotes/guides/security/jsse/JSSERefGuide.html#HowSSLWorks): "The client tells the server which cipher suites it has available, and the server chooses the best mutually acceptable cipher suite." – user207421 Apr 25 '12 at 12:56
  • 2
    And I found this: http://bugs.sun.com/bugdatabase/view_bug.do;jsessionid=3ea7ba5151d828fffffffff02e2b3e1a73b04?bug_id=4330535 It seems that the implementation is to honor preference order only for the client side. For the server they only pick the first enabled from the client. This is consistent with my experiments. So the JSSE reference seems also incorrect. And is too vague (the "best" is selected. Best in terms of what?) – Jim Apr 25 '12 at 13:10
  • @Jim Well there you go. Urban myths abound ;-) – user207421 Apr 26 '12 at 01:22
1

Something to realize: the client has a say in the selection also. If the client is saying it can only support the 128 bit cipher then that is what will be picked. See id your client even supports AES 256 by removing all supported ciphers but that one.

You are not providing a list of ciphers in order of preference. You are providing a list of acceptable ciphers that has to match one from the client. If any of those are not acceptable remove them.

Andrew T Finnell
  • 13,417
  • 3
  • 33
  • 49
  • I know that about handshake.The client does support the `AES 256` (send in `Client Hello`) and the ciphers I tested with are all supported by client.No preference order? SUN advertises a preference order http://docs.oracle.com/javase/6/docs/technotes/guides/security/SunProviders.html It says: `The table below shows the ciphersuites supported by SunJSSE in their default preference order`. Am I musunderstanding the meaning of "preference order" in their statement – Jim Apr 25 '12 at 05:49
  • No you are not mistaken. Notice that TLS_RSA_WITH_AES_128_CBC_SHA is ABOVE TLS_RSA_WITH_AES_256_CBC_SHA, hence it is picked first. – Andrew T Finnell Apr 25 '12 at 05:53
  • You are right. Somehow I didn't notice that. I noticed that if I remove `TLS_RSA_WITH_AES_128_CBC_SHA` from my list then the `256_CBC_SHA` is selected. I am wondering what is the meaning of `default` statement in `default preference order`. Could I configure a different preference order? I.e. configure the enabled ciphers and to pick e.g. the stronger or the `EC` before `DH` etc – Jim Apr 25 '12 at 06:22
  • From my experience, it is very much the clients cipher list that governs which one the server selects. As an example, my cipher list had ECDH ciphers at the top locations followed by SSL_RSA_WITH_AES_128_CBC_SHA256. However, because the certificate didn't specify EC parameters it always chose the RSA version. However, when I took everything else out apart from this and SSL_DHE_RSA_WITH_AES_128_CBC_SHA256 - putting the later first - it picked the later. So, from my experience it seems that order is important when it comes to the server making a decision. Cursed oracle/ibm ordering :P – andrewktmeikle Oct 23 '13 at 14:30
0

While your statement that 2246 (or 5256) does not say that the server picks the cipher suite from mutually supported cipher suites in the preference order supplied by the client is correct, NIST 800-57 Part 3, in section 4.3, Procurement Guidance, states:

  1. Implementations should select cipher suites in the order of preference submitted by the client.
Rick Ripberger
  • 9
  • 1