I have encountered a case where the client requires that I implement "mutual auth" by exchanging each others x509 public certs. Is this heard of?
I believe its still called mutual authentication.
Generally, certificate based mutual authentication falls into one of two models. The first is the enterprise model with a CA hierarchy, and the organization's CA signs both the client and server certificate.
The second model is clients using self-signed certificates in what is called Origin Bound Certificates. Its called Origin Bound because each site (origin) that needs a certificate gets the client to provide one. Origin bound certificates are the basis of the IETF's Token Binding protocol.
Its not clear to me if token binding has the same security properties as origin bound certificates. You see, we can use origin bound certificates to thwart man-in-the-middle attacks by making authentication part of channel setup. Token binding decouples the binding and moves it up in the stack. I believe it allows the MitM to act as an intermediary.
Enterprise Certificates
In the enterprise model, here's what you do with OpenSSL.
At the server perform the following. The server will handle the client certificate validation:
- Call
SSL_CTX_set_verify
with SSL_VERIFY_PEER
and SSL_VERIFY_FAIL_IF_NO_PEER_CERT
.
- Call
CTX_set_client_CA_list
to set the list of issuer CA the server will accept. This causes the appropriate SSL/TLS message to be sent to the client prompting for the certificate. This only sends a list of names to the client; they still must be trusted at the server
- Call
SSL_CTX_load_verify_locations
with issuers the server accepts. This adds the trust on the server side.
At the client, perform the following:
- Call
SSL_CTX_use_certificate_file
to load the client certificate
- Call
SSL_CTX_use_certificate_chain_file
as needed
- Call
SSL_CTX_use_PrivateKey
to load the private key
Origin Bound Certificates
Origin bound and self signed certificates are a little different because there's no CA hierarchy.
At the server perform the following. The server does not call SSL_CTX_load_verify_locations
or CTX_set_client_CA_list
because its a self signed certificate.
- Call
SSL_CTX_set_verify
with SSL_VERIFY_PEER
and SSL_VERIFY_FAIL_IF_NO_PEER_CERT
.
- Call
SSL_get_peer_certificate
after key exchange. Validate the client certificate presented to the server.
At the client, perform the following. Its the same as the enterprise model.
- Call
SSL_CTX_use_certificate_file
to load the client certificate
- Call
SSL_CTX_use_certificate_chain_file
as needed
- Call
SSL_CTX_use_PrivateKey
to load the private key
The lack of an enterprise CA means the client's self signed certificate needs to be communicated out-of-band to the server. Then the server needs to keep some sort of directory for looking up client certificates based on distinguished names and serial numbers. What you really care about here is the client's public key. In this case the X509 certificate is a presentation detail. Its just packaging because it usually binds the identity to a public key though an authority's signature (but not in this model).
The attack in this model - the 500-lb gorilla in the room - is the bad guy impersonating a user by using the same Distinguished Name and Serial Number because there is no Registration Authority (RA). You will need some measures to ensure you are not duped, like sending an email to a user to confirm their public key changes are expected.
The attack means that when you enroll a user, you need three or four things:
- Unique {Distinguished Name, Serial Number} pair
- Public key (part of the X509 certificate)
- A confirmation/recovery email address
To muddy the waters further, the user may have 3 or 4 devices, so they create a new origin bound certificate for each device. You will need to handle that enrollment gracefully, too.
To collectively take it full circle, what really matters is the email address and the different public keys/identities associated with the email address. The identities are housed in a X509 certificate with a unique {Distinguished Name, Serial Number} pair. You probably want them unique for auditing purposes, but there will likely be some copy/pasting going on among devices.
I am struggling to find any documentation or information on this with openssl.
You are probably having trouble finding information because this is a higher level, security architecture and design problem. To help with some references, start by reading Dietz, Czeskis, Balfanz and Wallach's Origin-Bound Certificates: A Fresh Approach to Strong Client Authentication for the Web. Also visit Peter Gutmann's Engineering Security and Ross Anderson's Security Engineering.
Origin Bound Certificates can be used to replace nearly every password based authentication system. It applies to nearly all the systems, from username/password based authentication to API keys used in web services. The password is still used to protect the local private key, but the password is not put on the wire.
When data sensitivity levels warrant stronger security controls, client certificates is one of the first things we turn to stop the mishandling of the password and botched authentication and authorization controls. Don't buy into Apple, Microsoft, Google (et al) use and handling of passwords. Its been defective for years. Its simply companies making it easy on users so they can capture business.