RSA OAEP Encryption with SHA-256 fails while with SHA-1 is OK

Question

I'm using Pkcs11Interop Library and trying to test encryption and decryption with RSA_PKCS_OAEP mechanism.

CK_RSA_PKCS_OAEP_PARAMS p = new CK_RSA_PKCS_OAEP_PARAMS();
p.HashAlg = (uint)CKM.CKM_SHA_1;
p.Mgf = (uint)CKG.CKG_MGF1_SHA1;
p.Source = (uint)CKZ.CKZ_DATA_SPECIFIED;
p.SourceData = IntPtr.Zero;
p.SourceDataLen = 0;

CK_MECHANISM mech = CkmUtils.CreateMechanism(CKM.CKM_RSA_PKCS_OAEP, p);

Everything is OK with the above mechanism but if I change the hash algorithm to SHA-256 like below:

CK_RSA_PKCS_OAEP_PARAMS p = new CK_RSA_PKCS_OAEP_PARAMS();
p.HashAlg = (uint)CKM.CKM_SHA256;
p.Mgf = (uint)CKG.CKG_MGF1_SHA256;
p.Source = (uint)CKZ.CKZ_DATA_SPECIFIED;
p.SourceData = IntPtr.Zero;
p.SourceDataLen = 0;

CK_MECHANISM mech = CkmUtils.CreateMechanism(CKM.CKM_RSA_PKCS_OAEP, p);

Then I get CKR_ARGUMENTS_BAD exception. I have been searching and debugging for a while but found nothing.

Have you tried to keep SHA-1 for MGF1 and only change the hash algorithm to SHA-256 since that does not have any adverse security effects? — Artjom B., Nov 18 '20 at 20:59

score 1 · Accepted Answer · answered Nov 18 '20 at 21:38

1

I had the same problem with Luna HSM (but was given CKR_MECHANISM_PARAM_INVALID).

That version of HSM simply did not support OAEP with SHA-256 and firmware upgrade was needed. After firmware upgrade it worked without any problems. Check if your device supports this variant.

Your code seems ok, I used (in java):

CK_RSA_PKCS_OAEP_PARAMS mechanismParams = new CK_RSA_PKCS_OAEP_PARAMS(
    CKM.SHA_1,
    CKG.MGF1_SHA1,
    new CK_RSA_PKCS_OAEP_SOURCE_TYPE(CKZ.DATA_SPECIFIED.longValue())
    , null, 0
);

and

CK_RSA_PKCS_OAEP_PARAMS mechanismParams = new CK_RSA_PKCS_OAEP_PARAMS(
    CKM.SHA256,
    CKG.MGF1_SHA256,
    new CK_RSA_PKCS_OAEP_SOURCE_TYPE(CKZ.DATA_SPECIFIED.longValue())
    , null, 0
);

Good luck!

answered Nov 18 '20 at 21:38

vlp

7,811
2
23
51

I have found out myself before seing this but you are right so I am accepting it. My HSM wasn't supporting OAEP with SHA256. – Ekrem Solmaz Nov 19 '20 at 09:38
could you let us know what version of firmware supports this, please – menakshisundaram Jan 26 '23 at 04:45
@menakshisundaram We upgraded from 6.10.2 to 6.27.0 (it was in 2019). – vlp Jan 26 '23 at 08:23

RSA OAEP Encryption with SHA-256 fails while with SHA-1 is OK

1 Answers1