As most other people have pointed out you have a problem which you are sending the most important part of an asymmetric key in clear text where the secret key can be intercepted. Sure you could encrypt the key, but that introduces other encryption problems to solve. If you use symmetric encryption algorithm to encrypt the key you have a key agreement problem to solve. How will each side agree on a key in a secure manner so that a 3rd party can't intercept it?
At some level you are re-solving problems SSL has already solved. Why not just use it? Also SSL has been battle tested and security experts have gone over it for 25 years. To develop your own version means you may fall into similar pitfalls that have already been encountered by SSL developers.
But, if you must re-implement you are going to need to change your algorithm. The most secure way to protect the private key is to never send it over the network in the first place. Asymmetric encryption allows you to trade messages between the client and server without the key agreement problem symmetric algorithms have.
Here's an algorithm that will get around the key agreement problem:
- Have the server generate an asymmetric key. Two options:
- Generate an asymmetric key one time (ie at server startup) and all connections use the same key.
- Generate an asymmetric key for each client when they connect making it more secure.
- When the client connects transfer the server's public key to the client
- The client will generate a secret key for a symmetric algorithm (How do you pick the algorithm? SSL solved this too).
- Client encrypts this secret key using the server's public key
- The client transfers the encrypted secret key to the server.
- The server decrypts the message using its private key to obtain the symmetric secret key.
- Now the server and client can exchange any encrypted message using symmetric encryption and a 3rd party can't listen in.
While even if you did this you still are open to man in the middle attacks where a 3rd party listener could trick the client and server to send unencrypted data through the 3rd party as the client trades messages with the client and server. Just a simple example of how implementing this stuff yourself could lead to you creating something less secure than just using SSL.
If you do send encrypted payloads around you'll want to ditch DataInputStream and go with straight binary streams so you don't need to convert binary payloads to UTF strings with Base64Encoding.