0

I have a high level question about public key authentication in SSL (doing a school project on openSSL). Anyway, as I understand it, this is basically how SSL works..

Server creates private key using RSA or DSA. Uses private key to create public key and adds public key to its certificate. Certificate is self signed or sent to a certificate authority to be digitally signed.

When a client wishes to connect to the server, it requests the certificate, checks to see if its trusted, then encrypts its message using the certificates public key. The message is sent to the server and decrypted using the private key.

Here is where my question comes in. How does the server securely respond? The public key is public so if it encrypts with the private key and expects the client to decrypt using the public key, someone in the middle could decrypt as well... the only solution I could see is if the roles reverse and the client becomes the server, and vise versa.

Anyone have some guidance here? I would appreciate it!

jww
  • 97,681
  • 90
  • 411
  • 885
theCamburglar
  • 224
  • 1
  • 2
  • 10
  • 4
    This question appears to belong on another site in the Stack Exchange network because its not about programming. Perhaps [Super User](https://www.superuser.com/), [Information Security Stack Exchange](https://security.stackexchange.com/) or [Cryptography Stack Exchange](https://crypto.stackexchange.com/). – jww May 12 '14 at 00:51
  • "When a client wishes to connect to the server, it requests the certificate, checks to see if its trusted, then encrypts its message using the certificates public key. The message is sent to the server and decrypted using the private key." - this varies from partially incorrect to totally incorrect (depending on the statement and some context). [Network Security: Private Communication in a Public World](http://www.amazon.com/dp/0130460192) has a very approachable explanation. Does your school library have it available? – jww May 12 '14 at 00:56
  • @jww I agree, I had already flagged it for migration. – Maarten Bodewes May 12 '14 at 12:28

2 Answers2

1

The public key can be used to encrypt a pre master secret (in TLS versions before 1.3), which can be decrypted by the server. A set of symmetric keys - for instance AES keys - are are then derived from this pre-master secret. This key can then be used for the subsequent information. This is also called hybrid encryption because it both involves asymmetric (public/private key) and symmetric (single secret key) encryption.

Alternatively the private key can be used to authenticate using an authentication protocol that includes Ephemeral Diffie-Hellman. Diffie-Hellman is an asymmetric algorithm (just like RSA) that can be used to agree on a the pre-master secret without an eavesdropper gaining knowledge of the key. Ephemeral means that the Diffie-Hellman keys are not persistent, in other words, the complete DH protocol is executed each time. In this case RSA is used to authenticate using a signature instead of encryption. This has protocol has the property that the SSL protocol cannot be decrypted even when the private key gets exposed later on (this is called forward secrecy).

So the answer is that RSA is not used to encrypt all the data, it is only used in the process to establish the symmetric session keys. Those keys are used to encrypt (and protect for integrity and authenticity) the data in the follow-up messages. Note that it depends on the selected cipher suite if RSA is used at all in SSL/TLS.

Maarten Bodewes
  • 90,524
  • 13
  • 150
  • 263
0

You're right. Anyone can decrypt messages encrypted by the server(private key), because they have the corresponding public key. However, there is some level of security here because only the corresponding public key can decrypt data encrypted by a private key. Hence the client can be sure that the message really is sent by the server, because only the server has the private key(ideally).
Additionally, the TLS/SSL protocol does not just rely on assymetric encryption(it is computationally expensive), it is used for key exchange, to exchange data used to derive the keys to be used for the session. The actual keys are never transmitted.

automaton
  • 1,091
  • 1
  • 9
  • 23
  • 1
    "*Anyone can decrypt messages encrypted by the server(private key), because they have the corresponding public key.*". That's why "scrambling stuff" with a private key isn't called encrypting, but signing. – Bruno May 13 '14 at 10:23
  • Semantics, but important note nonetheless. – automaton May 15 '14 at 12:51