2

I want to sign a SHA-256 hash with DSA using PKCS#11 Java Wrapper for the PKCS#11 API of a hardware security module. For this purpose I selected the Mechanism CKM_DSA, load the corresponding DSA key from the token and have the data (read as byte-array) signed. The key I use for testing has 1024bit length.

Everything seems to work fine: The key is loaded, the Session.sign() yields a byte[] array of length 40. This corresponds to the PKCS#11 Spec, which says:

"For the purposes of this mechanism, a DSA signature is a 40-byte string, corresponding to the concatenation of the DSA values r and s, each represented most significant byte first."

Now I want to verify this signature using openSSL, i.e., using

openssl dgst -d -sha256 -verify ${PUBLIC_KEY} -signature signature.der <raw input file>

This works if I

a) created the signature using OpenSSL

b) created the signature using bouncycastle and encoding the result as ASN1 encoded DER sequence.

Now I want to do the same with the PKCS#11 signature. My question is: how to format this 40 byte array? I tried the following:

        //sign data
        byte[] signedData = this.pkcs11Session.sign(dataToSign);
        //convert result
        byte[] r = new byte[20];
        byte[] s = new byte[20];
        System.arraycopy(signedData, 0, r, 0, 20);
        System.arraycopy(signedData, 19, s, 0, 20);

        //encode result
        ASN1EncodableVector v = new ASN1EncodableVector();
        v.add(new ASN1Integer(r));
        v.add(new ASN1Integer(s));
        return new DERSequence(v).getEncoded(ASN1Encoding.DER);

The encoding part seems to be correct, because it works if I produce r and s directly with bouncycastle and another software key. Besides, openssl does accept the input format but the verification fails sometimes with an error, sometimes just with "Verification failure".

Thus, I assume the conversion of the PKCS#11 signature to r and s is wrong. Can someone help finding the mistake?

  • The comments were removed, including the one where I asked for a full stack trace. Please provide the full stack trace and preferably some test data and keys. – Maarten Bodewes Mar 04 '17 at 14:46
  • Shouldn't the second array copy start at pos 20 instead of 19? Seems like you have the last byte of r twice, and don't ever get the last byte of s. – bartonjs Mar 05 '17 at 18:33

1 Answers1

3

You probably have to convert the r and s values to a BigInteger class before you do. The reason for this is that ASN.1 uses signed value encoding and DH results in unsigned value encoding. So you've got a pretty high chance of getting a negative value in your ASN.1, which will result in an error.

To perform the conversion, use new BigInteger(1, r) and new BigInteger(1, s) and put the result into the ASN1Integer instances. Here 1 indicates that the value needs to be converted to a positive value (i.e. the input is unsigned positive).

Maarten Bodewes
  • 90,524
  • 13
  • 150
  • 263
  • LuxOnion wrote: "Thanks for your suggestion. I tried to convert r and s to BigInteger. Both, as BigInteger(r) or - as you suggest - BigInteger(1,r) (or, s, respectively). I observe, when parsing the ASN1Sequence with OpenSSL that values are always positive (and I agree, I would expect according to DSA spec that s cannot be negative). However, still the verification fails." – Maarten Bodewes Mar 04 '17 at 14:45