Encryption in Java with RSA/ECB/OAEPWithSHA-256AndMGF1Padding could not be decrypted in .NET

Question

We have a requirement where in Java does the encryption and .NET does the decryption. The issue is with Asymmetric decryption in .NET, the algorithm, mode and padding used in Java while encryption is "RSA/ECB/OAEPWithSHA-256AndMGF1Padding". On .NET, I have tried to decrypt in all different ways RsaCng, Rsa, Chilkat Rsa, Bouncy Castle with OAEP padding, nothing worked so far. Below are the code snippets. Only when Java's encryption approach is changed to "RSA/ECB/OAEPWithSHA-1AndMGF1Padding", .NET was able to decrypt, but due to security reasons, Java can't change the approach of encryption. We are using PEM file to decrypt, while using a CERT to encrypt. Please let us know the best way to fix the issue. Is it something to do with creating a PEM file which is SHA256 hash based? I would appreciate if someone can help me out.

Errors are mentioned below. RsaCng and Rsa:- Parameter is Incorrect. RsaCryptoService:- Error occurred while decoding OAEP padding. Bouncy Castle:- It was giving a alpha-unicode string with symbols. Chilkat:- Returning null.

Java Snippet for Asymmetric Encryption:-

                 // Encrypt the symetric key
                 X509Certificate cert = ConduentCertificateLocator.getConduentCertificate();
                 PublicKey certKey = cert.getPublicKey();
                 Cipher c2 = Cipher.getInstance("RSA/ECB/OAEPWITHSHA-256ANDMGF1PADDING");
                 //Cipher c2 = Cipher.getInstance("RSA");
                 c2.init(Cipher.ENCRYPT_MODE, certKey);
                 byte[] inputCertKey = key.getEncoded();
                 byte[] encryptedKey = c2.doFinal(inputCertKey);
                 String encryptedKeyUrl = encodeToURL(encryptedKey);

.NET Snippet for Asymmetric Decryption:- We did try in few different ways. Below is the code.

a. RSACryptoService

           using (RSACryptoServiceProvider csp = new RSACryptoServiceProvider())
            {
                using (var stream = File.OpenRead(_appSettings.PEM))
                using (var reader = new PemUtils.PemReader(stream))
                {
                    var rsaParameters = reader.ReadRsaKey();
                    csp.ImportParameters(rsaParameters);

                    var resultBytes = csp.Decrypt(ksBytes, true);
                    finalResult = Convert.ToBase64String(resultBytes);
                }
            }

b. RsaCng

           using (RSA rsa = new RSACng())
            {
                using (var stream = File.OpenRead(_appSettings.PEM))
                using (var reader = new PemUtils.PemReader(stream))
                    rsa.ImportParameters(reader.ReadRsaKey());
                byte[] decrypted2 = rsa.Decrypt(ksBytes, RSAEncryptionPadding.OaepSHA256);
                var ss = Encoding.UTF8.GetString(decrypted2);
            }

c. Rsa (System.Security.Cryptography)

          using (RSA rsa = RSA.Create())
            {
                using (var stream = File.OpenRead(_appSettings.PEM))
                using (var reader = new PemUtils.PemReader(stream))
                {
                    var rsaParameters = reader.ReadRsaKey();

                    rsa.ImportParameters(rsaParameters);
                }


                finalResult = rsa.Decrypt(ksBytes, RSAEncryptionPadding.OaepSHA256);
            }

d. Chilkat

       using (PrivateKey pkey = new PrivateKey())
        {
            pkey.LoadPemFile(_appSettings.PEMFile);
            key = pkey.GetXml();
        }

        using (Chilkat.Rsa decryptor = new Chilkat.Rsa())
            {
                decryptor.ImportPrivateKey(key);

                // Make sure we have the same settings used for encryption.
                decryptor.OaepPadding = true;
                decryptor.EncodingMode = "hex";
                decryptor.OaepHash = "sha-256";

                string eStr = decryptor.DecryptStringENC(ks, true);
            }

e. BouncyCastle

        AsymmetricCipherKeyPair keyPair;
        using (var reader = File.OpenText(_appSettings.PEM))
            keyPair = (AsymmetricCipherKeyPair)new 
           Org.BouncyCastle.OpenSsl.PemReader(reader).ReadObject();
          var decryptEngine = new OaepEncoding(new RsaBlindedEngine(), new Sha256Digest(), new 
               Sha1Digest(), null);
            decryptEngine.Init(false, keyPair.Private);
            var decrypted = Encoding.UTF8.GetString(decryptEngine.ProcessBlock(ksBytes, 0, 
             ksBytes.Length));

Answer 1

My scenario is a little different but it should help you. I needed to decrypt in java and it was encrypted in .NET. So the opposite of the question proposed, but like the others have mentioned. Java underneath the hood defaults to SHA-1 underneath the scenes even if you specify RSA/ECB/OAEPWithSHA-256AndMGF1Padding . The trick for me is the pass one extra parameter telling it to actually use SHA-256.

JAVA:

private static byte[] decryptBytes(byte[] buffer, Key key) throws InvalidKeyException, NoSuchPaddingException, NoSuchAlgorithmException, BadPaddingException, IllegalBlockSizeException, InvalidAlgorithmParameterException {
    Cipher rsa;
    rsa = Cipher.getInstance("RSA/ECB/OAEPWithSHA-256AndMGF1Padding");
    rsa.init(Cipher.DECRYPT_MODE, key, new OAEPParameterSpec("SHA-256", "MGF1", new MGF1ParameterSpec("SHA-256"), PSource.PSpecified.DEFAULT));
    return rsa.doFinal(buffer);
}

.NET

var encryptedBytes = publicKey.Encrypt(bytesToEncrypt, RSAEncryptionPadding.OaepSHA256);