SSLEngine: Invalid TLS padding data when unwrapping after a successful handshake

Question

I am using non-blocking socketChannel and SSLEngine an ssl server. So after a successfull handshake, i read the socket ( 184 bytes /384 bytes are being read in the first time), and then i pass this buffer to unwrap method. The unwrap method throw the following exception :

javax.net.ssl.SSLHandshakeException: Invalid TLS padding data
    at sun.security.ssl.Alerts.getSSLException(Unknown Source)
    at sun.security.ssl.SSLEngineImpl.fatal(Unknown Source)
    at sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(Unknown Source)
    at sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
    at javax.net.ssl.SSLEngine.unwrap(Unknown Source)

But in case I read all the bytes (384/384) in the first time, then I don't get this exception.

I thought that if the sslengine doesn't have enough byte to unwrap it will return a bufferUnderflow status.

Do I really need all bytes to call unwrap method? If yes how can I be sure I read all bytes for the non blocking socket?

---

EDIT: the code:

public boolean doHandShake(SocketChannel socket) throws Exception{

        if(!socket.isConnected()){
            return false;
        }

        outAppData.clear();
        inAppData.clear();
        inNetData.clear();
        outNetData.clear();

        if(engine==null || socket==null)
         return false;


          engine.beginHandshake();
          SSLEngineResult.HandshakeStatus hs = engine.getHandshakeStatus();

          while (hs != SSLEngineResult.HandshakeStatus.FINISHED &&
                    hs != SSLEngineResult.HandshakeStatus.NOT_HANDSHAKING) {

              switch (hs) {

              case NEED_UNWRAP:
                   int read=1;
                  while (read > 0) {
                            read=socket.read(inNetData);
                            if(read==-1){
                                throw new IOException ("channel closed");
                            }
                        }

                  inNetData.flip();
                  engineRes=engine.unwrap(inNetData, outAppData);
                  inNetData.compact();

                  switch(engineRes.getStatus()){
                            case BUFFER_OVERFLOW:
                                System.out.println("overFlow");
                                break;
                            case CLOSED:
                                return false;
                            case OK:
                                //outAppData.clear();
                            //  inNetData.clear();
                                break;
                            default:
                                break;
                  }

              break;

              case NEED_WRAP :
                 outNetData.clear();
                  engineRes=engine.wrap(inAppData, outNetData);
                  outNetData.flip();
                  switch (engineRes.getStatus()){
                            case BUFFER_OVERFLOW:
                                System.out.println("overFlow");
                                break;
                            case BUFFER_UNDERFLOW:
                                System.out.println("underFlowa");
                                break;
                            case CLOSED:
                                return false;
                            case OK:
                                //outNetData.flip();
                                while(outNetData.hasRemaining()){
                                    if(socket.write(outNetData)<0){
                                        throw new Exception("Channel Has been Closed");
                                    }
                                }

                                break;
                            default:
                                break;


                  }

              break;

              case NEED_TASK :
                  Runnable r=engine.getDelegatedTask();
                  r.run();
                  break;

              case FINISHED:
                  System.out.println("finished");
                    break;

              case NOT_HANDSHAKING:
                    break;

                  default: 
                      throw new IllegalArgumentException("Inexpected/Unhadled SSLEngineResult :"+hs);

              }

              hs = engine.getHandshakeStatus();

          }
          return true;

    }

then I read 184/384 bytes using non blocking channel.

read = _socketChannel.read(buffer);

and then pass the to buffer to be decrypted:

public ByteBuffer decrypt(ByteBuffer inNetData) throws SSLException{

        if(!isValidSession()){
            return null;
        }
            outAppData.clear();

            try{
              engineRes=engine.unwrap(inNetData, outAppData);
            }catch(Exception e){
                e.printStackTrace();
            }
              inNetData.compact();

              switch(engineRes.getStatus()){
                    case BUFFER_OVERFLOW:
                        outAppData=ByteBuffer.allocate(outNetData.capacity()*2);
                        inNetData.position(0);
                        return encrypt(inNetData);
                    case BUFFER_UNDERFLOW:
                        return null;
                    case CLOSED:
                        return null;
                    case OK:
                        outAppData.flip();
                        System.out.println(new String(outAppData.array(),0,400));

                        return outAppData;
                    default:
                        break;
              }


        return null;
    }

the exception is being thrown in engine.unwrap engineRes=engine.unwrap(inNetData, outAppData);

Answer 1

Several issues here.

1. If you get BUFFER_OVERFLOW you must flip(), write(), compact(), and repeat the wrap(). Not just print it and give up as above.

2. If you get BUFFER_UNDERFLOW you must read() and repeat the unwrap(). This is your specific problem.

3. If you get an Exception of any kind it isn't sufficient to just print the exception and continue as though it hadn't happened.

4. FINISHED is never returned by getHandshakeStatus(). It is only ever set in the handshake status in the SSLEngineResult that is returned by wrap() and unwrap().

5. You are calling encrypt() from within the decrypt() method. This doesn't make sense.

6. Your overall technique won't work. A handshake can happen any time, in the middle of a read or write, and you have to write your code so as to cope with that. When you've succeeded in that, you won't need to call your own doHandshake() method at all, as your code will cope with it anyway.

The SSLEngine is a state machine. You must do exactly what it tells you to do.