0

I'm using a RC4 stream cipher to encrypt/decrypt data sent through a Java UDP client-server program. My code works as intended on the host program and displays the correct plaintext and ciphertext, but when the data is sent over to the client via UDP, the ciphertext in particular get a super long ciphertext appended to it.

The client is still able to decrypt it properly, I just don't understand where the random appended ciphertext is coming from.

Host program:

public class Host {
    
    private static final String ENCRYPTION_ALGORITHM = "ARCFOUR"; // or "RC4"

    public static void main(String[] args) throws Exception {
        
        //Use Java's built-in DatagramSocket and DatagramPacket to implement UDP
        DatagramSocket socket = new DatagramSocket(1500); //Create a socket to listen at port 1500
        byte[] buf = new byte[65535]; //Byte array to wrap
        
        //Initialize P, G, H(PW), A variables
        BigInteger P = null;
        BigInteger G = null;
        String hashedPW = null; //Hashed PW using SHA-1
        BigInteger A = new BigInteger(10, new Random()); //Random number selected by Alice, maximum limit of 1000
        BigInteger aliceKey = null;
        
        //Read in Diffie-Hellman parameters from param.txt
        File file = new File("param.txt");
        Scanner scan = new Scanner(file);
        
        //Loop through the entire file
        while(scan.hasNextLine()) {
            int counter = 0;
            String delimiter = ","; //Content in the file is separated using a comma

            String[] tokenValue = scan.nextLine().split(delimiter);
                for (String token : tokenValue) {
                    switch (counter++) {
                        case 0: //Diffie-Hellman's P
                            P = new BigInteger(token);
                            break;
                        case 1: //Diffie-Hellman's G
                            G = new BigInteger(token);
                            break;
                        case 2: //Password hashed using SHA-1
                            hashedPW = token;
                            break;
                    }
                }
            }
        scan.close(); //Close scanner to prevent memory leaks
        
        aliceKey = G.modPow(A, P); //Calculate g^a mod p to obtain Alice's public key
        
        byte[] passwordBytes = hashedPW.getBytes();
        
        //Setting up RC4 stream cipher
        SecretKey secretKey = new SecretKeySpec(passwordBytes, 0 , passwordBytes.length, "RC4"); //Generate secret key using common password
        Cipher rc4 = Cipher.getInstance(ENCRYPTION_ALGORITHM);
        
        System.out.println("Parameters successfully read. Listening on port 1500...");

        //While-loop to keep host running until terminated
        while (true) {
          DatagramPacket message = new DatagramPacket(buf, buf.length); //Create a packet to receive message
          socket.receive(message); //Receive the message from Bob
          
          //If Alice receives a packet with the message "Bob"
          if(new String(message.getData(),0,message.getLength()).equals("Bob")) {
              System.out.println("Bob has sent a connection request.");
              
              String msgToBob = hashedPW + "," + P + "," + G + "," + aliceKey; //H(PW), P, G, G^A mod P
              message.setData(encrypt(msgToBob, secretKey, rc4)); //Encrypt data using RC4
              
              System.out.println("Ciphertext sent: " + new String(message.getData(),0,message.getLength()));
              System.out.println("Decrypted text: " + decrypt(secretKey, rc4, message.getData()));
              
              socket.send(message);
          }
        }
    }

Client program:

public class Client {
    
    private static final String ENCRYPTION_ALGORITHM = "ARCFOUR"; // or "RC4"

    public static void main(String[] args) throws Exception {
        
        //Use Java's built-in DatagramSocket and DatagramPacket to implement UDP
        DatagramSocket socket = new DatagramSocket(); //Create socket object to send data
        byte[] buffer = new byte[65535];
        socket.setSoTimeout(5000); //Throw an exception if no data received within 5000ms
        
        //Create scanner to grab user input
        Scanner input = new Scanner(System.in);
        System.out.print("Please enter the common password: ");
        String commonPW = input.nextLine(); //Storing the password

        commonPW = encryptThisString(commonPW); //Hash using SHA-1
        byte[] passwordBytes = commonPW.getBytes(); //Convert password to byte-array
        
        //Setting up RC4 stream cipher
        SecretKey secretKey = new SecretKeySpec(passwordBytes, 0 , passwordBytes.length, "RC4"); //Generate secret key using common password
        Cipher rc4 = Cipher.getInstance(ENCRYPTION_ALGORITHM);
        
        System.out.print("Enter a message to send to Alice: ");
        String bobInput = input.nextLine();
        
        //Create packet containing message
        DatagramPacket message = new DatagramPacket(buffer, buffer.length, InetAddress.getByName("localhost"), 1500);
        message.setData(bobInput.getBytes());
        socket.send(message); //Send message to host
        
        //Create separate packet to receive response (so that its length is big enough to accept)
        DatagramPacket response = new DatagramPacket(buffer, buffer.length);
        socket.receive(response);
        
        System.out.println("Received from Alice: " + new String(response.getData(),0,response.getLength()));
        System.out.println("Decrypted text: " + decrypt(secretKey, rc4, response.getData()));
        
        socket.close();
    }

And here's the same methods in both programs used to encrypt/decrypt data:

private static byte[] encrypt(String plaintext, SecretKey secretKey, Cipher rc4) throws InvalidKeyException, IllegalBlockSizeException, BadPaddingException {
          rc4.init(Cipher.ENCRYPT_MODE, secretKey);
          byte[] plaintextBytes = plaintext.getBytes();
          byte[] ciphertextBytes = rc4.doFinal(plaintextBytes);
          //System.out.println("RC4 ciphertext base64 encoded: " + Base64.encodeBase64String(ciphertextBytes));
          return ciphertextBytes;
      }

      private static String decrypt(SecretKey secretKey, Cipher rc4, byte[] ciphertextBytes) throws InvalidKeyException, InvalidAlgorithmParameterException, IllegalBlockSizeException, BadPaddingException {
          rc4.init(Cipher.DECRYPT_MODE, secretKey, rc4.getParameters());
          byte[] byteDecryptedText = rc4.doFinal(ciphertextBytes);
          String plaintextBack = new String(byteDecryptedText);
          return plaintextBack;
      }

The console output for host:

Parameters successfully read. Listening on port 1500...

Bob has sent a connection request.

q?Md|-A????±?h??♥?a????h¶^O eÆá<±?ÿ.♂/?9§}5?aaJj??fë???7óIp§T▬??ñk??lümµÅ⌂»??=L?rúk?§}x☻?.☻j:A·æu?8ë∟2äq*k∟♥?C?$??VZIù?!0|?á=I?X{♥²"‼°▲|c??2²ht??²☻P=?T?"?,9?ƒZ▼?X??M=d uöåä$M☼ñ?"♂♀ö«2&??SC?î?Å?|?#?B??⌂b9?♀??T?lƃî D^í?àêoO3{N3♦^L²¼?=ë??-?|?O{???B?♣ #Æ??½.ìV:ó?±]Ü4▲?|rA£jEGJé??♠?$K"'=«??D»m←ÿ4?J↕¥§y¶[£↨?U¬y-?ÿpëñù«←x→↨[²äueN???↨}☻zƒy??si??z?U[q??Q ?+?÷¡,7Ü?S{6?l???²??↨?%?+üá??? '½?J??«?¶?T.±@=Q!ó??>½ëY2?P☻a♂**?iLO???☺⌂♫?ñ=N?ö?k$4??äîóí←?si?↓ìÜg∟Z6↕?µáù½Rû▲b?9µ,???g?6ù?→nv?ûyµö?M¿U♥â? e[?V?Zô$¿?¢ôq♥j????G??|?~0ÇGZ?z♥68&MoK??v·$<?âZ^??Ys??[g?? ½å\▬ß¿?·↔!N???.?'?AD?Vg?ën¢-6ô ???0,ö???}lä??Y?D?Xñ¬?º??²☺r¥ï??1?#pq?►p?▬??FMÜ-m>ßñè,?¡r??µ±@?C?¶£?¿→?oi O??♥óeòM£?N ë_?▲èv¡ï?Ü(?öW♥w²ù]q???e♠à♠íô?í^??QM?^?5éÑ??V(??? V?>è?à%?D-r∟Ü1??
N↑??·*?A?tTßæi?[??ú¬c¢ƒl??↓▲ÿie→?♠tL¡??'Ñe÷∟ ƒí?♠↑4 Ä?R;}?:'¡!n6◄▲ ☻'??r?I???↨¬z?→XÄ??☼Ä{ñ:â⌂$ê{????àß?FRí???☻ª3?#9‼/¿?$/ï?{ç
öûu½Rz?▼?ô,??◄K[P∟►§PS??))?â?d▲í♥?9??Ü?¡? ««+?☼☺?<<°n¼?♂%óî↔úög

Decrypted text: 62a91401ef53536f5e4a282ca75ce2505b7f8dfe,133369106234410027239548844977121529173266794839390492022055432118963330212454946189658890281117740419231366102782626716256746218028068963594259491015958826219687167085932845323487318056081767660358673429760060452507829995471137991271021787608959982667244524929925606086596447565032707095625628293300812984091,15635507126291796032175381295261956049808038047523713881770866167494167302066120851790523759612392465402233123769308400911864440257625348397936936978922581645934310469147780992255584968935730234568508327975497520199946653504137607154138430285573487119797346741445139288140860073660395495603540644148828628224,96950246366083756365146382678616471348261490848687079335489625506702434220187716384031671233394087317377823909861019537627642960196032766557467991541000545780675424518198042877768900211198791646583260247888450303207379340549434158094990174523043323964083031376798844776809887666868545542060025549549427788304

Output for Client is similar except for the decrypted plaintext:

Please enter the common password: Popcorn

Enter a message to send to Alice: Bob

q?Md|-A????±?h??♥?a????h¶^O eÆá<±?ÿ.♂/?9§}5?aaJj??fë???7óIp§T▬??ñk??lümµÅ⌂»??=L?rúk?§}x☻?.☻j:A·æu?8ë∟2äq*k∟♥?C?$??VZIù?!0|?á=I?X{♥²"‼°▲|c??2²ht??²☻P=?T?"?,9?ƒZ▼?X??M=d uöåä$M☼ñ?"♂♀ö«2&??SC?î?Å?|?#?B??⌂b9?♀??T?lƃî D^í?àêoO3{N3♦^L²¼?=ë??-?|?O{???B?♣ #Æ??½.ìV:ó?±]Ü4▲?|rA£jEGJé??♠?$K"'=«??D»m←ÿ4?J↕¥§y¶[£↨?U¬y-?ÿpëñù«←x→↨[²äueN???↨}☻zƒy??si??z?U[q??Q ?+?÷¡,7Ü?S{6?l???²??↨?%?+üá??? '½?J??«?¶?T.±@=Q!ó??>½ëY2?P☻a♂**?iLO???☺⌂♫?ñ=N?ö?k$4??äîóí←?si?↓ìÜg∟Z6↕?µáù½Rû▲b?9µ,???g?6ù?→nv?ûyµö?M¿U♥â? e[?V?Zô$¿?¢ôq♥j????G??|?~0ÇGZ?z♥68&MoK??v·$<?âZ^??Ys??[g?? ½å\▬ß¿?·↔!N???.?'?AD?Vg?ën¢-6ô ???0,ö???}lä??Y?D?Xñ¬?º??²☺r¥ï??1?#pq?►p?▬??FMÜ-m>ßñè,?¡r??µ±@?C?¶£?¿→?oi O??♥óeòM£?N ë_?▲èv¡ï?Ü(?öW♥w²ù]q???e♠à♠íô?í^??QM?^?5éÑ??V(??? V?>è?à%?D-r∟Ü1??
N↑??·*?A?tTßæi?[??ú¬c¢ƒl??↓▲ÿie→?♠tL¡??'Ñe÷∟ ƒí?♠↑4 Ä?R;}?:'¡!n6◄▲ ☻'??r?I???↨¬z?→XÄ??☼Ä{ñ:â⌂$ê{????àß?FRí???☻ª3?#9‼/¿?$/ï?{ç
öûu½Rz?▼?ô,??◄K[P∟►§PS??))?â?d▲í♥?9??Ü?¡? ««+?☼☺?<<°n¼?♂%óî↔úög

Decrypted text: 62a91401ef53536f5e4a282ca75ce2505b7f8dfe,133369106234410027239548844977121529173266794839390492022055432118963330212454946189658890281117740419231366102782626716256746218028068963594259491015958826219687167085932845323487318056081767660358673429760060452507829995471137991271021787608959982667244524929925606086596447565032707095625628293300812984091,15635507126291796032175381295261956049808038047523713881770866167494167302066120851790523759612392465402233123769308400911864440257625348397936936978922581645934310469147780992255584968935730234568508327975497520199946653504137607154138430285573487119797346741445139288140860073660395495603540644148828628224,96950246366083756365146382678616471348261490848687079335489625506702434220187716384031671233394087317377823909861019537627642960196032766557467991541000545780675424518198042877768900211198791646583260247888450303207379340549434158094990174523043323964083031376798844776809887666868545542060025549549427788304?úmB½Ä????←ê???►\d⌂h♦4LM♣☼»??*Ñ?ì?çtº???ì?.◄☺????xK♥??Ä4öó↑↨ 4?á☼mëQ«a

The actual output is much longer for the client program, I didn't copy the entire appended ciphertext. Not sure if it's relevant, but the console makes some Windows notification/error noise when it runs as well, although it doesn't actually give any compilation errors. I'm running my program using command prompt.

I don't think the issue lies with the decryption as I tested with some text on both the host and client programs and I managed to get the same decrypted plaintext just fine. Only when I send the data over UDP it causes this issue it seems.

The text infront of the original ciphertext such as "Ciphertext sent: " gets omitted in the output as well.

What am I missing?

Zeffia
  • 37
  • 5
  • If I'm reading the code correctly, you always send the entire 65535-byte `buffer` regardless of how much content you actually wanted to send. – teapot418 Feb 03 '23 at 10:14
  • Altering the size of the byte buffer on the client side does seem to change the decrypted output length, but I'm not sure how to do it to make it fit the size of the response from host... – Zeffia Feb 03 '23 at 10:39
  • 1
    You're trying to decrypt the entire packet buffer instead of that fraction of it that contains the actual message. See my comment [HERE](https://stackoverflow.com/questions/75333814/extra-characters-appearing-after-rc4-decryption) – g00se Feb 03 '23 at 12:17
  • @g00se Yeah I figured that out myself after much time spent messing around. Appreciate the help though! – Zeffia Feb 03 '23 at 13:22
  • Then you should have processed the question instead of letting people like me waste their time – g00se Feb 03 '23 at 14:22

1 Answers1

1

I was trying to decrypt the entire packet buffer instead of that fraction of it that contains the actual message, which resulted in the large wall of text after the actual decrypted message.

Altering the decrypt function used to instead take in the appropriate packet size fixed the issue.

Zeffia
  • 37
  • 5