SSLEngine giving NEED_UNWRAP after unwrapping server hello done

Question

I am using DTLS1.0 provided by java 9. It successfully generate Client Hello and server response back with

 1. Server Hello
 2. Certificate
 3. Server Key Exchange
 4. Certificate Request
 5. Server Hello Done

Then SSLEngine gives NEED_UNWRAP. After unwrapping the packets containing the Server Hello Done it again gives NEED_UNWRAP. After unwrapping the next re transmitted Server Hello Done it again gives NEED_UNWRAP. It goes again and again. But i think it should generate the next handshaking signal by giving NEED_WRAP.

If i am wrong, please correct me. Otherwise Why it is happening?

Trust Manager:

    final TrustManager[] trustAllCerts = new TrustManager[] {
        new X509TrustManager() {

            public X509Certificate[] getAcceptedIssuers() {
                // TODO Auto-generated method stub
                return null;
            }

            public void checkServerTrusted(X509Certificate[] arg0, String arg1)
                    throws CertificateException {
                // TODO Auto-generated method stub

            }

            public void checkClientTrusted(X509Certificate[] arg0, String arg1)
                    throws CertificateException {
                // TODO Auto-generated method stub

            }
        }
    };

SSLEngine:

        char[] passphrase = "123456".toCharArray();//This is the password

        // First initialize the key and trust material
        KeyStore ksKeys = KeyStore.getInstance("JKS");
        ksKeys.load(new FileInputStream("keystore"), passphrase);

        // KeyManagers decide which key material to use
        KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
        kmf.init(ksKeys, passphrase);


        SSLContext sslContext = SSLContext.getInstance("DTLSv1.0");
        sslContext.init(kmf.getKeyManagers(), trustAllCerts, null);


        int port2 = Queuemanager.switchMediaHandler.get("192.168.19.148").realPort;
        // Create the engine
        engine = sslContext.createSSLEngine("192.168.19.148", port2);

        // Use as client
        engine.setUseClientMode(true);
        engine.setEnableSessionCreation(true);

Handshake:

    void doHandshake(){
    engine.beginHandshake();        
    SSLEngineResult result; 
    HandshakeStatus handshakeStatus;        
    int appBufferSize = engine.getSession().getApplicationBufferSize();
    ByteBuffer myAppData = ByteBuffer.allocate(appBufferSize);
    ByteBuffer peerAppData = ByteBuffer.allocate(appBufferSize);
    myNetData.clear();
    peerNetData.clear();

    handshakeStatus = engine.getHandshakeStatus();
    while (handshakeStatus != SSLEngineResult.HandshakeStatus.FINISHED && handshakeStatus != SSLEngineResult.HandshakeStatus.NOT_HANDSHAKING) {             
        switch (handshakeStatus) {
        case NEED_UNWRAP_AGAIN:
            logger.debug("NEED_UNWRAP_AGAIN");
        case NEED_UNWRAP:
            logger.debug("NEED_UNWRAP");
            DatagramPacket packet = null;
           if(handshakeStatus == SSLEngineResult.HandshakeStatus.NEED_UNWRAP ) {
            try {
                byte[] buf = new byte[1024];
                packet = new DatagramPacket(buf, buf.length);
                socket.receive(packet);
                peerNetData = ByteBuffer.wrap(buf, 0, packet.getLength());
                peerAppData =  ByteBuffer.allocate(appBufferSize);
            } catch (IOException e1) {
                // TODO Auto-generated catch block
                e1.printStackTrace();
            }
          }else{
               peerNetData = ByteBuffer.allocate(0);
               peerAppData =  ByteBuffer.allocate(appBufferSize);
           }


            SSLEngineResult.Status rs;
            result = null;
            try {

                handshakeStatus = engine.getHandshakeStatus();
                while(handshakeStatus == SSLEngineResult.HandshakeStatus.NEED_UNWRAP || handshakeStatus == SSLEngineResult.HandshakeStatus.NEED_UNWRAP_AGAIN) {
                    result = engine.unwrap(peerNetData, peerAppData);
                    peerNetData.compact();  
                    handshakeStatus = result.getHandshakeStatus();
                 }

                handshakeStatus = result.getHandshakeStatus();

                rs = result.getStatus();

            } catch (SSLException sslException) {
                engine.closeOutbound();                    
                break;
            }
            switch (rs) {
            case OK:
                break;
            case BUFFER_OVERFLOW:
                break;
            case BUFFER_UNDERFLOW:
                break;
            case CLOSED:
            default:
                throw new IllegalStateException("Invalid SSL status: " + result.getStatus());
            }
            break;
        case NEED_WRAP:
            logger.debug("NEED_WRAP");
            myNetData.clear();
            try {
                result = engine.wrap(myAppData, myNetData);
                handshakeStatus = result.getHandshakeStatus();
            } catch (SSLException sslException) {
                engine.closeOutbound();
                handshakeStatus = engine.getHandshakeStatus();
                break;
            }
            switch (result.getStatus()) {
            case OK :                                       

                while (myNetData.hasRemaining()) {
                    //String str = myNetData.toString();
                    byte[] arr = new byte[myNetData.remaining()];
                    myNetData.get(arr);
                    recvPacket = new DatagramPacket(arr, arr.length);
                    recvPacket.setData(arr);

                    try {
                        int port2 = Queuemanager.switchMediaHandler.get("192.168.19.148").realPort;

                        InetAddress ip = InetAddress.getByName("192.168.19.148");       

                        recvPacket.setAddress(ip);
                        recvPacket.setPort(port2);     

                        socket.send(recvPacket);

                    } catch (IOException e) {
                        // TODO Auto-generated catch block
                        e.printStackTrace();
                    }
                    //socketChannel.write(myNetData);
                }

            case BUFFER_OVERFLOW:
            case BUFFER_UNDERFLOW:
            case CLOSED:
            case NEED_TASK:
                 Runnable task;
                 while ((task = engine.getDelegatedTask()) != null) {
                      new Thread(task).start();
                 }
                 handshakeStatus = engine.getHandshakeStatus();
                 break;
    }           
}

user207421 · Accepted Answer · 2018-04-16T07:21:28.893

1

You're doing this back to front. When the engine says NEED_UNWRAP, do the unwrap. Then, if that returns BUFFER_UNDERFLOW, do the read and try the unwrap again. Don't just assume you need another read on NEED_UNWRAP. You may be losing data from the peerNetData the way you're doing it. And don't create a new peerNetData via ByteBuffer.wrap() or .allocate() either: keep using the same one, and use put() to put the datagram data into it. Then compact() it after every successful engine unwrap.

In other words, do exactly and only what it tells you to do.

The SSLEngine is a very difficult thing to get right. Many have tried: few have succeeded. For one working success, that is the basis for a commercial product, see the SSLEngineManager class in the source code for my book Fundamental Networking in Java, Springer 2006, here.

edited Apr 16 '18 at 07:21

answered Apr 03 '18 at 02:41

user207421

305,947
44
307
483

Thank you for your reply. Do i need to call unwrap again with the previous peerNetData or i need to clear peerNetData and then call unwrap? – Rashed Apr 03 '18 at 03:37
I have updated the code according to your suggestion. Now after couple of unwrap it gives NEED_TASK. After handling that it gives NEED_UNWRAP_AGAIN, then if i try to unwrap it gives exception: javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2. plz have a look. – Rashed Apr 03 '18 at 03:52
You should *never* clear `peerNetData`, and you should never create a new one either via `ByteBuffer.wrap()`. You should `compact()` it after every unwrap. The engine will advance the `position` over data it has processed. I don't see where you have done what I stated in my answer. I don't know what NEED_UNWRAP_AGAIN, is but you don't need it. The status is still NEED_UNWRAP from when you last got it. It isn't a new state in any way. – user207421 Apr 03 '18 at 04:54
This is getting too broad to answer further but I've answered similar questions on it before several times. – user207421 Apr 03 '18 at 05:06
Ok. Thank you for quick response. Plz let me try your suggestions. – Rashed Apr 03 '18 at 05:07
I am sorry for that. I am new to this field. And i am really stuck. That's why i posted it. – Rashed Apr 03 '18 at 05:08
No problem, it's a very difficult thing to get right. Many have tried: few have succeeded. – user207421 Apr 03 '18 at 05:09
If you don't mind, Can you plz give any working example link? – Rashed Apr 03 '18 at 05:18
Done, see edit. – user207421 Apr 03 '18 at 05:28

SSLEngine giving NEED_UNWRAP after unwrapping server hello done

1 Answers1