Signature verification failure due to reordering via _asn1_ordering_set_of

Question

I am using GnuTLS 3.4.1. I have a x509 certificate with set of sequences inside. The certificate is stored that way on a smart card.

GnuTLS is rearranging the sequences via function _asn1_ordering_set_of, which appears to be causing a verification failure.

Here's what the sequence looks like:

SEQUENCE : 
...
   SET : 
      SEQUENCE : 
         OBJECT_IDENTIFIER : 'CN (2.5.4.3)'
         PrintableString : '0000'
      SEQUENCE : 
         OBJECT_IDENTIFIER : 'SN (2.5.4.4)'
         TeletexString : 'XXX'
      SEQUENCE : 
         OBJECT_IDENTIFIER : 'G (2.5.4.42)'
         TeletexString : 'YYY'

OpenSSL (and probably Java PKCS11 provider) loads this construction as is. GnuTLS on load of the certificate sorts this construction in _asn1_ordering_set_of. So that it becomes:

SEQUENCE : 
...
   SET : 
      SEQUENCE : 
         OBJECT_IDENTIFIER : 'G (2.5.4.42)'
         TeletexString : 'YYY'
      SEQUENCE : 
         OBJECT_IDENTIFIER : 'SN (2.5.4.4)'
         TeletexString : 'XXX'
      SEQUENCE : 
         OBJECT_IDENTIFIER : 'CN (2.5.4.3)'
         PrintableString : '0000'

Why does GnuTLS sort the set of sequences? What way should it be done, is it a GnuTLS bug or other libraries simply omit ordering?

*Why does GnuTLS sort the set of sequences* - probably it is a preparation for DER serialization. — mkl, May 12 '15 at 19:57
@elevener - what is the URL to retrieve a certificate that's failing GnuTLS verification? — jww, May 13 '15 at 20:56

Kurt Roeckx · Answer 1 · 2015-05-13T21:40:52.303

3

RFC5280 has:

4.1. Basic Certificate Fields

The X.509 v3 certificate basic syntax is as follows. For signature calculation, the data that is to be signed is encoded using the ASN.1 distinguished encoding rules (DER) [X.690].

So it seems to me that GnuTLS is doing the right thing.

It also looks like it's trying to encode a Distinguished Name, but does it wrong. It's valid according to the ASN1, because the spec is just really weird. You can have multiple values for each part. But you want the CN, SN and so on each in it's own SET, so all those SEQUENCEs should have had their own SET.

edited May 13 '15 at 21:40

answered May 13 '15 at 20:47

Kurt Roeckx

173
1
5

Thanks Kurt. I was not aware of that. Two questions: (1) is there a way to relax GnuTLS restrictions so it accepts BER (due to the signature); and (2) should a bug report be filed with OpenSSL to promote interop (and compliance)? For (1), the `tbsCertificate` should be DER. I'm not sure about the leap that signed data should be reordered. – jww May 13 '15 at 21:00
Can someone send the certificate to kurt@roeckx.be? – Kurt Roeckx May 13 '15 at 21:51
Kurt - this is kind of an interesting issue from the policy/procedure side of things. I asked on the PKIX mailing list to get some feedback on verification (GnuTLS and you are clearly right about DER and signing the `tbsCertifcate`). See [Signature Verification and DER requirements](https://www.ietf.org/mail-archive/web/pkix/current/msg33308.html). – jww May 13 '15 at 22:05
Certificate is stored that way on a smart card. I'll learn a bit later if I am allowed to send it. But to be honest (I don't know x509 encoding standards, that's why I asked) I don't get why GNUTLS reorders SEQUENCES inside of SET and even more from what I see reorders them in descending order – elevener May 14 '15 at 10:30
sent the cerificate on email – elevener May 14 '15 at 14:27
@elevener - could you share that with everyone? I know two other libraries that could use it as a test case :) If so, paste the PEM encoding into the question. – jww May 14 '15 at 20:05
The sorting is not based on the string attribute, but on the DER encoding of the whole sequence. – Kurt Roeckx May 15 '15 at 11:20

jww · Answer 2 · 2015-05-13T21:02:04.753

What way should it be done ...

The ITU recommends SET OF be encoded using BER, as long as there's no need for CER or DER. The best I can tell, there's no need. See below for a more detailed explanation in the realm of the ITU and ASN.1.

However, GnuTLS may be complying with a standard that creates the need. In this case, I'm not aware of which standard it is. See Kurt's answer.

~~I looked at RFC 5280, PKIX Certificate and CRL Profile, but I could not find the restriction. Maybe its in another PKIX document.~~

is it a GnuTLS bug or other libraries simply omit ordering?

I don't believe its a bug in GnuTLS per se. Its just the way the library does things. Take this modulo the requirement to do so in a RFC or other standard.

Also note that other libraries don't omit ordering. They use the order the attributes are presented in the certificate, which is an ordering :)

(comment) The problem is that GNUTLS rearranging results in failed SSL authentication

That sounds like a bug to me (modulo standards requirements). In this case, the bug is reordering the SET OF after a signature is placed upon the TBS/Certificate.

If GnuTLS is building the TBS/Certificate, then its OK to reorder until the signature is placed upon it.

(comment) Does GnuTLS put the elements of a SET OF type in the correct order according to DER rules

In ASN.1 encoding rules, X.690, BER/CER/DER:

8.12 Encoding of a set-of value
...
8.12.3 The order of data values need not be preserved by the encoding and subsequent decoding.

A SET OF does not appear to be ordered (for example, lexicographical order), so the sender can put them in any order, and a receiver can reorder them.

However, 11.6 says:

11 Restrictions on BER employed by both CER and DER
...
11.6 Set-of components
The encodings of the component values of a set-of value shall appear in ascending order, the encodings being compared as octet strings with the shorter components being padded at their trailing end with 0-octets. NOTE – The padding octets are for comparison purposes only and do not appear in the encodings.

In the above, they are saying BER can be any order, but CER and DER are ascending order.

And last but not least, the Introduction says:

Introduction
...
... The basic encoding rules is more suitable than the canonical or distinguished encoding rules if the encoding contains a set value or set-of value and there is no need for the restrictions that the canonical and distinguished encoding rules impose ...

So the Introduction recommends BER for SET OF.

But in the big picture: the certificate is in BER. That's how it was signed. GnuTLS cannot change that once they get a hold of the certificate because of the signature over the certificate's data.

GnuTLS is free to create certificates in DER encoding. They just can't impose the encoding after the fact.

(comment) gnutls_certificate_set_x509_key_file(xcred, CERT_URL, KEY_URL, GNUTLS_X509_FMT_PEM);

I looked at the latest GnuTLS sources. That's appears to be the way its used in src/serv.c.

Apparently, _asn1_ordering_set_of was not working as expected in the past. It was improved in April, 2014. See PATCH 1/3: Make _asn1_ordering_set_of() really sort (and friends) on the GnuTLS mailing list.

Here are the hits for it in the sources:

$ grep -R -n _asn1_ordering_set_of * | grep -v doc
lib/minitasn1/coding.c:832:  /* Function : _asn1_ordering_set_of */
lib/minitasn1/coding.c:843:  _asn1_ordering_set_of (unsigned char *der, int der_len, asn1_node node)
lib/minitasn1/coding.c:1261:  err = _asn1_ordering_set_of (der + len2, counter - len2, p);

The use around line 1261 is for asn1_der_coding. asn1_der_coding is used more frequently in other components...

(comment) but I'm not sure that it's bug in GNUTls and not on the server side, so I'd like to find out how it should work before doing anything

You should probably reach out ot the GnuTLS folks as detailed at B.3 Bug Reports. It looks like a bug in the processing of non-GnuTLS certificates.

To be clear, GnuTLS uses DER when it creates certificates and that's fine. But GnuTLS cannot impose ordering after it receives a non-GnuTLS certificate because that invalidates the signature.

Their test suite probably misses it because GnuTLS DER encodes SET OF. They likely are not aware its happening.

The problem is that GNUTLS rearranging results in failed SSL authentification. So the question is GNUTls right and it "puts the elements of a SET OF type in the correct order according to DER rules" or there is no such rules for SEQUENCES and it does it on it's own accord. — elevener, May 12 '15 at 16:00
@elevener - that sounds like a bug to me. Just in case, you should show some of your code around the verification. You might also add that fact to your question since it seems to be an important fact :) — jww, May 12 '15 at 16:10
gnutls_certificate_set_x509_key_file(xcred, CERT_URL, KEY_URL, GNUTLS_X509_FMT_PEM); and perform TLS handshake. With commented out _asn1_ordering_set_of connection is established ok, with original version fails. Last version of GNUTls 3.4.1 from the site (do you know if patch you mentioned made it to the distribution). It also looks like bug to me, but I'm not sure that it's bug in GNUTls and not on the server side, so I'd like to find out how it should work before doing anything. — elevener, May 13 '15 at 09:16
According to more recent editions of X.509, I think GnuTLS is wrong. There's text which explicitly says "When checking signatures in received data, it shall check the signature against the actual data received rather than its conversion of the received data to a distinguished encoding." That text is from X.509 11/2008. (I'm fairly sure that text was not present in '93.) — Bruce Stephens, May 14 '15 at 16:10
@Bruce - I did not check X.509 on the subject. But I probably should have since its an obvious choice in hindsight. — jww, May 14 '15 at 18:05
@Bruce - possibly you had better comment Kurt Roeckx answer? (or I can copy your comment there) I think he is one of the maintainer of GNUTLS, so he is the person to be persuaded :) — elevener, May 18 '15 at 16:09

Signature verification failure due to reordering via _asn1_ordering_set_of

2 Answers2