Decrypting RSA encrypted data from C# in Java. Format of encrypted data?

Question

I am using RSA encryption to encrypt some data in C#. Now I want to decrypt the encrypted data in Java. But I've encountered some problems.

The main problem probably is to get the encrypted message from c# to java. In c# we have unsigned bytes and the endian is different

So, for testing I am converting the byte array of the encrypted data in c# to a sbyte array and get a string representation of it. Then I copy the string representation of the byte array into my java code and convert it back to an 'byte' array. After that, I reverse the array to match the endianess of java.

But if I try to decode the data that was like above transferred, I get following exception:

javax.crypto.BadPaddingException: Message is larger than modulus

Encrypting and decrypting from C# to C# works as well as from java to java. only C# to java won't work. (the string to encrypt has a length of 7 characters, so it is not really to long)

I am converting the public key in c# to a BigInteger. The public key is delivered from the RSAParameters:

 public byte[] RSAEncrypt(byte[] data, RSAParameters param, bool padding) {
        using (RSACryptoServiceProvider rsaProvider = new RSACryptoServiceProvider(1024)) {
            rsaProvider.ImportParameters(param);

            byte[] modulusArray = param.Modulus;
            byte[] exponentArray = param.Exponent;

            BigInteger modulusBigInt = new BigInteger(modulusArray);

            BigInteger exponentBigInt = new BigInteger(exponentArray);

            encryptedData = rsaProvider.Encrypt(data, false);

            return encryptedData;
        }
    }

After that I copy the string representations of the modulus and exponent into my java code and create new BigInteger from them and create the public key:

    BigInteger modulusBigInt = new BigInteger(modulusBytesStr);
    BigInteger exponentBigInt = getBigIntFromByteString(exponentBytesStr);

    Key pK = getPublicKey(modulusBigInt, exponentBigInt);

Then I try to decrypt the data (where data is the byte array that I transferred from c# to java like described above):

        Cipher cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");

        cipher.init(Cipher.DECRYPT_MODE, pK);

        decryptedData = cipher.doFinal(data);

But if i try so, I am getting the exception mentioned above. The public key should be right, I think. At least I have the same BigInteger values of the modulus and the exponent in c# and in java. The padding is equal, too. So I am assuming that there is something wrong with the format of my encrypted data. Which fromat should it have?

I have also read this question: RSA .NET encryption Java decryption But even after that I am not sure what format my data to encryspt/decrypt should have

EDIT: tried to convert the encrypted bytes in c# to a Base64 String and in java converting it back to bytes. Also not working

EDIT2: If I am using var key = rsaProvider.ToXmlString(true); to get the xml represantation of the public key and putting the xml strings of the modulus and exponents in my java code, converting them from Base64 string to byte array and from byte array to BigInteger, then I am getting another value of the BigInteger of the modulus as the BigInteger had in c#, but I get follwing exception with this values: javax.crypto.BadPaddingException: Decryption error

EDIT3: Found my error: For simple testing I just used the private key that I generated in my C# code to decrypt in java. But in my java code I tried to generate a public key from the private key.

     RSAPublicKeySpec keySpec = new RSAPublicKeySpec(modulus, exponent);

        KeyFactory kf = KeyFactory.getInstance("RSA");
        PublicKey pK = kf.generatePublic(keySpec);

Thats clearly wrong. So I changed this to:

 RSAPrivateKeySpec keySpec = new RSAPrivateKeySpec(modulusBigInt, exponentBigInt);

        KeyFactory kf = KeyFactory.getInstance("RSA");
        Key key = kf.generatePrivate(keySpec);

And it worked. Thats also why the method of GregS worked (there was no 'PublicKey'/'PrivateKey' generated and for the decryption was not the java build in method used)

Answer 1

I'm a little unclear as to exactly where you are going wrong. Here is an example that works for me that I based off of your code. I don't have a windows machine so I tested this in Mono.

C#:

using System;
using System.Security.Cryptography;
using System.Text;

namespace RsaDotNetToJava
{
    class MainClass
    {
        public byte[] RSAEncrypt(byte[] data, RSAParameters param, bool padding) {
            using (RSACryptoServiceProvider rsaProvider = new RSACryptoServiceProvider(1024)) {
                rsaProvider.ImportParameters(param);

                var encryptedData = rsaProvider.Encrypt(data, false);

                return encryptedData;
            }
        }

        public static void W(string label, byte [] x) {
            var b64 = Convert.ToBase64String (x);
            Console.WriteLine ("{0} = {1}", label, b64);
        }

        public static void Main (string[] args)
        {
            var x = new MainClass ();
            var rsa = new RSACryptoServiceProvider (1024);
            var data = Encoding.ASCII.GetBytes("Hello world");
            var parms = rsa.ExportParameters(true);
            W ("Modulus", parms.Modulus);
            W ("P", parms.P);
            W ("DecryptExponent", parms.D);
            W ("EncryptExponent", parms.Exponent);
            var cipher = x.RSAEncrypt(data, parms, false);
            W ("Cipher", cipher);
        }
    }
}

Next, some Java that reads in the base64 strings with ("label = " stripped off) from the command line and does some manipulations. This uses the java 7 class javax.xml.bind.DatatypeConverter for its base64 parser.

import java.math.BigInteger;
import javax.xml.bind.DatatypeConverter;

public class RsaJavaToDotNet {


    private static BigInteger b64ToBigInteger(String b64) {
        byte [] bigEndianBytes = DatatypeConverter.parseBase64Binary(b64);
        return new BigInteger(1, bigEndianBytes); 
    }

    /**
     * @param args base64 encoded .NET big-endian integer arrays
     *    args[0] = modulus
     *    args[1] = prime
     *    args[2] = decrypt exponent
     *    args[3] = encrypt exponent
     *    args[4] = cipher
     * @throws Exception 
     */
    public static void main(String[] args) throws Exception {

        BigInteger modulus = b64ToBigInteger(args[0]);
        final int modulusByteLength = (modulus.bitLength() + 7) / 8;
        BigInteger prime = b64ToBigInteger(args[1]);
        BigInteger d = b64ToBigInteger(args[2]);
        BigInteger e = b64ToBigInteger(args[3]);
        BigInteger cipherInt = b64ToBigInteger(args[4]);

        // Decrypt the cipher

        BigInteger plainInt = cipherInt.modPow(d, modulus);
        byte [] plain = plainInt.toByteArray();

        // Verify the format and extract the message.

        if (plain.length != (modulusByteLength - 1) || plain[0] != 2) {
            throw new Exception("Something is wrong");
        }

        // Find the zero byte delimited the payload from the padding

        int zeroPos = 1;
        while (zeroPos < plain.length && plain[zeroPos] != 0) {
            ++zeroPos;
        }

        String plainStr = new String(plain, zeroPos + 1, plain.length - zeroPos - 1, "UTF-8");
        System.out.println(plainStr);
    }

}

Answer 2

So, for testing I am converting the byte array of the encrypted data in c# to a sbyte array and get a string representation of it. Then I copy the string representation of the byte array into my java code and convert it back to an 'byte' array. After that, I reverse the array to match the endianess of java.

I don't quite get why "endianness" is an issue here. Endianism is a function of the CPU you're running on, not the programming language.