4

I am trying to communicate from an iOS application to a Java backend. When using the .rsaEncryptionOAEPSHA256 algorithm in SecKeyCreateEncryptedData for encryption, the backend shows the following stacktrace:

Caused by: java.lang.SecurityException: Exception thrown while invoking doFinal(byte[]).
    at  ... 101 common frames omitted
Caused by: javax.crypto.BadPaddingException: Decryption error
    at java.base/sun.security.rsa.RSAPadding.unpadOAEP(RSAPadding.java:497)
    at java.base/sun.security.rsa.RSAPadding.unpad(RSAPadding.java:292)
    at java.base/com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:366)
    at java.base/com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:392)
    at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2207)
    ... 103 common frames omitted

The Java backend is using the RSA/ECB/OAEPWithSHA-256AndMGF1Padding algorithm from the SunJCE provider. This provider uses SHA-1 for the MGF1 digest. However the default for iOS is to use SHA-256.

  • How can I change this in the code for iOS to make this compatible?

Note: To be clear, the exact same interaction is currently working fine with .rsaEncryptionPKCS1 on iOS where the backend uses RSA/ECB/PKCS1Padding in Java.

mahler
  • 526
  • 5
  • 25
  • I could update the Java cipher to use `MGF1ParameterSpec.SHA256`, but unfortunately that isn't supported by Android, so I would rather re-configure the OAEP parameter spec on iOS. – mahler Sep 03 '20 at 17:40
  • Just out of curiosity, have you tried `.rsaEncryptionOAEPSHA1`? Not sure if that sets the MGF or only the hash function for the message... – Bram Dec 30 '20 at 01:30
  • .rsaEncryptionOAEPSHA1 - doesn't work – Leonif Dec 13 '22 at 09:34

1 Answers1

1

Faced the same issue, from what I've researched & experimented with, it's practically impossible to support RSA/ECB/OAEPWithSHA-256AndMGF1Padding on iOS.

Similiar issue: Encrypt RSA/ECB/OAEPWithSHA-256AndMGF1Padding Swift

But this only addressed the algorithm but not the padding issue.

samcro
  • 11
  • 1