I'm not sure if it's even possible. Also, OpenSSL is one ugly motherlover of an utility :/
I need top upload certificate+private key as DER to ESET Security Management Center (ESMC), at least according to their technical support.
I use XCA for this small deployment, and there's no option to export public+private as DER, only PEM (or a few other formats). So the question is: how to convert PEM private+public to DER?
The difference between PEM and DER is only the way how it is stored. DER is binary form and PEM is base64 encoded file (with "header"). As DER is more or less raw form you can store just one object in the file.
In case you need more objects (e.g. key and cert) you need PEM encoded form where based on the "header" is possible to differentiate the objects (in the file the objects follow each other) or some specific container format like PKCS12 (also supported by openssl).
For openssl you can use options inform and outform to specify if you are interested in PEM (default so used in case you don't request DER) or DER.
For the key (let assume rsa) - as PEM is default following commands are equal:
openssl rsa -in <file_with_key> -out <new_der_key_file> -outform DER
openssl rsa -in <file_with_key> -inform PEM -out <new_der_key_file> -outform DER
For the certificate - also two equal forms :
openssl x509 -in <cert_file> -out <new_der_cert> -outform DER
openssl x509 -in <cert_file> -inform PEM -out <new_der_cert> -outform DER
In case of pkcs12 container (if supported the transformation would be done during import if needed):
openssl pkcs12 -export -in <cert_file> -inkey <key_file> -out <pkcs12_file>.p12
To my knowledge, you can't store DER-encoded key and certificate in one file. You need to export the key and the certificate separately. Using XCA, you can do this, selecting the "DER" option at export.
If you use OpenSSL, you need to specify the outform switch, which dictates the format OpenSSL should use when writing the files (pem or der):
openssl rsa -in /the/cert/and/the/key.pem -out key_in_der.key -outform der
openssl x509 -in /the/cert/and/the/key.pem -out cert_in_der.crt -outform der
If you need to upload one file, then you need to use some kind of container (PKCS12 for example).
