decryption error occurring when using rsa in java.

Question

My aim is to take in a message containing a challenge and an origin. On receiving this msg a rsa keypair must be generated which is then used to manipulate the data as shown below. Certain part of the data is encrypted using the generated public key. During authorization, that data must be decrypted with the private key. However, when i try to decrypt it is shows a decryption error. I have even printed different parts of the code just to check if the desired output is achieved which is why i know the private key taken from file is correct. I am unable to solve the decryption error. The specifications for the task require the use of rsa and not hybrid encryption. i have tried padding but that didnt help. please advice on how to solve this problem

package pam;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Paths;

import java.security.*;
import java.security.spec.InvalidKeySpecException;
import java.security.spec.PKCS8EncodedKeySpec;
import java.security.spec.X509EncodedKeySpec;

import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.KeyGenerator;
import javax.crypto.SecretKey;
import javax.xml.bind.DatatypeConverter;
import com.sun.jersey.core.util.Base64;

class Test 
{
  public static void kpgen(int numBits, String s) throws Exception
  {
    if(s.length()!=64)
    {
        System.out.println("invalid entry");
    }
    else
    {
        KeyPairGenerator keygen = KeyPairGenerator.getInstance("RSA");
        KeyPair keyPair = keygen.genKeyPair();
        PrivateKey privateKey = keyPair.getPrivate();
        PublicKey publicKey = keyPair.getPublic();
        System.out.println("pk: "+privateKey);
        System.out.println("pubk: "+publicKey);

        String fileBase = "f:\\tempKey";        //WRITING PVT KEY TO FILE
        try (FileOutputStream out = new FileOutputStream(fileBase + ".key")) 
        {
            out.write(keyPair.getPrivate().getEncoded());
        }
        try (FileOutputStream out = new FileOutputStream(fileBase + ".pub")) 
{
            out.write(keyPair.getPublic().getEncoded());
        }

        System.out.println("Key pair : " + Base64.encode(String.valueOf(keyPair)));
        MessageDigest md = MessageDigest.getInstance("SHA-256");
        byte[] digest = md.digest(keyPair.toString().getBytes(StandardCharsets.UTF_8));
        String sha256 = DatatypeConverter.printHexBinary(digest).toLowerCase();
        System.out.println("Hash value: "+sha256);

        String ch = s.substring(0,32);
        String or = s.substring(32,64);
        System.out.println("Challenge: "+ch);
        System.out.println("Origin: "+or);

        MessageDigest md1 = MessageDigest.getInstance("SHA-256");
        byte[] digest1 = md1.digest(privateKey.toString().getBytes(StandardCharsets.UTF_8));
        String sha = DatatypeConverter.printHexBinary(digest1).toLowerCase();
        or = or + sha;
        System.out.println("String kh: "+or);

        Cipher cipher = Cipher.getInstance("RSA");  
        cipher.init(Cipher.ENCRYPT_MODE, publicKey);
        byte[] keyHandle = cipher.doFinal(or.getBytes());
        System.out.println("Key Handle: "+keyHandle);

        String f = "f:\\keyList.pub";
        Key pub = getKeyFromFile(f);
        System.out.println("Attestation Public Key: "+pub);

        PrivateKey pk = (PrivateKey) getPvtKey("f:\\keyList.key");
        Signature rsa = Signature.getInstance("SHA1withRSA"); 
        rsa.initSign(pk);
        rsa.update(ch.getBytes());
        byte[] sc = rsa.sign();
        System.out.println("Signed challenge: "+sc);

        String rm = publicKey.toString() + pub + sc + keyHandle;
        System.out.println("Response Msg: " +rm);
    }
}
public static Key getKeyFromFile(String fileName) throws Exception
{
    byte[] bytes = Files.readAllBytes(Paths.get(fileName));
    X509EncodedKeySpec ks = new X509EncodedKeySpec(bytes);
    KeyFactory kf = KeyFactory.getInstance("RSA");
    PublicKey pub = kf.generatePublic(ks);
    return pub;
}
public static PrivateKey getPvtKey(String s) throws IOException, NoSuchAlgorithmException, InvalidKeySpecException
{
    byte[] bytes = Files.readAllBytes(Paths.get(s));
    PKCS8EncodedKeySpec ks = new PKCS8EncodedKeySpec(bytes);
    KeyFactory kf = KeyFactory.getInstance("RSA");
    PrivateKey pvt = kf.generatePrivate(ks);
    return pvt;
}
public static void auth(String s) throws NoSuchAlgorithmException, Exception, IOException
{   
    String chal = s.substring(0, 32);
    String origin = s.substring(32,64);
    String kh = s.substring(64);
    byte[] kh1 = kh.getBytes();

    PrivateKey pvtKey = getPvtKey("f:\\tempKey.key"); //READING THE PRIVATE KEY MADE IN KPGEN
    System.out.println("pk: "+pvtKey);
    Cipher cipher = Cipher.getInstance("RSA");  
    cipher.init(Cipher.DECRYPT_MODE, pvtKey);
    byte[] keyHandle = cipher.doFinal(kh1);

    String or = keyHandle.toString().substring(0, (kh.length()/2));
    String pk = keyHandle.toString().substring(kh.length()/2);

    int c = 0;
    if(or.equals(origin))
    {
        c++;
    }
    else
    {
        System.out.println("Bad Key Handle: Invalid Origin");
    }
    if(pk.equals(pvtKey.toString()))
    {
        c++;
    }
    else
    {
        System.out.println("Bad Key Handle: invalid private key");
    }
    if(c==2)
    {
        Signature rsa = Signature.getInstance("SHA1withRSA"); 
        rsa.initSign((PrivateKey) pvtKey);
        rsa.update(chal.getBytes());
        byte[] sc = rsa.sign();
        System.out.println("Signed Challenge: "+sc);
    }
    else
        System.out.println("error");
}

}

score 2 · Answer 1 · answered Oct 22 '18 at 09:45

2

You have multiple (many) issues in your code with the encryption

First - encode properly your data, String in Java is to represent printable characters. As soon as you work with encryption (working on byte[] level), you need to encode or decode the values.

Example - your code will print the "keyHandle", it's a byte array object hash, not really the encrypted data itself

byte[] keyHandle = cipher.doFinal(or.getBytes());
System.out.println("Key Handle: "+keyHandle);
...
String rm = publicKey.toString() + pub + sc + keyHandle;

Use at hex or base64 encoding to print out the output. The same applies to the signature.

I am unable to solve the decryption error.

String kh = s.substring(64);
byte[] kh1 = kh.getBytes();
..  
byte[] keyHandle = cipher.doFinal(kh1);

And you simply assume you can decrypt some random substring? Encrypting using RSA will produce output of size of the key (e.g. 2048 bits) and you have to store and decrypt as whole, not any substring.

As a learning exercise - try to simply encrypt and decrypt, encode, decode to learn the primitives you can (re)use.

answered Oct 22 '18 at 09:45

gusto2

11,210
2
17
36

byte[] kh = Base64.decode(s); Cipher cipher = Cipher.getInstance("RSA"); cipher.init(Cipher.DECRYPT_MODE, pvtKey); byte[] keyHandle = cipher.doFinal(kh); – Namit Mehra Oct 23 '18 at 11:25
the above comment is the code im using now. it isnt working in this program and gives decryption error. if i try it separately it works. in above code 's' is the complete input value – Namit Mehra Oct 23 '18 at 11:37
@NamitMehra `s` is assumed to be whole encrypted value (output of encryption). if it works separately, super. It means you provide different value in your program – gusto2 Oct 23 '18 at 11:54
ok. so if i am understanding correctly it is somehow getting the wrong key or a wrong input? also, padding is not the issue right? – Namit Mehra Oct 23 '18 at 12:14
Padding is applied before encryption and checked during decryption (by the cipher implementation itself). So yes, you either provide different key or different input ("s" must be whole and exact output of the previous encryption) – gusto2 Oct 23 '18 at 12:45

decryption error occurring when using rsa in java.

1 Answers1