How to programmatically create encrypted PKCS#7 using ECC certificate and OpenSSL in C?

Question

OpenSSL console command to do this is:

openssl cms -encrypt -aes-256-cbc -in plain-original.txt -outform der -out encrypted.p7 -recip certificate.pem -keyopt ecdh_kdf_md:sha256

assuming certificate.pem is EC type. I'm trying with CMS_encrypt function (this is working for RSA)

STACK_OF(X509)* stackOfX509 = sk_X509_new_null();
X509* x509Cert = d2i_X509(NULL, &(cert->buff), cert->buffLen);
sk_X509_push(stackOfX509, x509Cert);
CMS_ContentInfo* pkcs7EnvelopedData = CMS_ContentInfo_new();
BIO* dataBIOToEncrypt = BIO_new_mem_buf(data2Encrypt->buff, data2Encrypt->buffLen);

pkcs7EnvelopedData = CMS_encrypt(stackOfX509, dataBIOToEncrypt, cipher, CMS_BINARY);

This of course dosen't work. I have no idea how to pass keyopt. I'm looking for this for a while now. Found i.e EVP_PKEY_CTX_ctrl_str(ctx, "ecdh_kdf_md", "sha256"); but this requires me to have EVP_PKEY_CTX.

Before I start digging through OpenSSL source code to determine how they do it, maybe someone can give me any hint?

--- UPDATE ---

Ok, I know now I did oversimplified my first attempt. Getting closer I hope:

pkcs7EnvelopedData = CMS_encrypt(NULL, dataBIOToEncrypt, cipher, CMS_BINARY | CMS_PARTIAL);
CMS_RecipientInfo *ri = CMS_add1_recipient_cert(pkcs7EnvelopedData, x509Cert, CMS_BINARY | CMS_PARTIAL | CMS_KEY_PARAM);
EVP_PKEY_CTX *pctx = CMS_RecipientInfo_get0_pkey_ctx(ri);
EVP_PKEY_CTX_ctrl_str(pctx, "ecdh_kdf_md", "sha256");
EVP_CIPHER_CTX *wctx = CMS_RecipientInfo_kari_get0_ctx(ri);
EVP_EncryptInit_ex(wctx, EVP_aes_256_wrap(), NULL, NULL, NULL);
CMS_final(pkcs7EnvelopedData, dataBIOToEncrypt, NULL, CMS_BINARY | CMS_PARTIAL);

Shane Powell · Accepted Answer · 2021-04-08T11:33:58.240

If you breakdown the openssl application cms.c source file to just generate your command what you get is:

bool CMS_encrypt_example(char const* in_file_name, char const* out_file_name, char const* recip_cert_path )
{
    // openssl cms -encrypt -aes-256-cbc -in plain-original.txt -outform der -out encrypted.p7 -recip certificate.pem -keyopt ecdh_kdf_md:sha256

    auto in = make_handle(BIO_new_file(in_file_name, "rb"), BIO_free);
    if(!in) return false;

    auto file = make_handle(BIO_new_file(recip_cert_path, "r"), BIO_free);
    auto cert = PEM_read_bio_X509(file.get(), nullptr, nullptr, nullptr);
    if(!cert) return false;

    auto flags = CMS_PARTIAL;
    auto const content_info = make_handle(CMS_encrypt(nullptr, in.get(), EVP_aes_256_cbc(), flags), CMS_ContentInfo_free);
    if(!content_info) return false;

    auto* ri = CMS_add1_recipient_cert(content_info.get(), cert, flags | CMS_KEY_PARAM);
    if(ri == nullptr) return false;

    auto* pctx = CMS_RecipientInfo_get0_pkey_ctx(ri);
    if (EVP_PKEY_CTX_ctrl_str(pctx, "ecdh_kdf_md", "sha256") <= 0) return false;

    if(!CMS_final(content_info.get(), in.get(), nullptr, flags)) return false;

    auto const outfile = make_handle(BIO_new_file(out_file_name, "wb"), BIO_free);
    if(!outfile) return false;
    if(i2d_CMS_bio_stream(outfile.get(), content_info.get(), in.get(), flags) == 0) return false;

    return true;
}

You will have to forgive me C++ usage since you asked for C but the above example compiles for me and generates the correct output as per your example openssl command.

The only real difference to your code is that it's calling i2d_CMS_bio_stream to write the output instead of you trying to use the CMS_final which the docs says that parameter is only used for detached data and you aren't doing that here.

Also your code uses the CMS_BINARY option, which would add the -binary option to your openssl command.

Yes, I also did use `cms.c` as a point of reference. You did also execute `CMS_final` so the code seems identical, except I don't need file handler. Also `EVP_EncryptInit_ex` seems unnecessary. I'm building this on iOS and it's crashing at `CMS_final` somwhere in `CMS_dataInit: crypto/evp/evp_enc.c`, but I'm not good at reading ASM. I'm guessing bad input parameters for algorithms maybe. So, I think I will go debugging app with the command line to determine inputs. Anyway thank you for answer! That got me on right track :) — Krzysztof Derecki, Apr 08 '21 at 11:08
On the other hand your code compiles/runs as you say, so they should be good. This must be something else. — Krzysztof Derecki, Apr 08 '21 at 11:21
Are you checking your return codes to see one of your previous to the final call hasn’t failed? One possible problem may be the type of cert you add in due to the ecdh requirement makes the EVP_PKEY_CTX_ctrl_str call fail. — Shane Powell, Apr 08 '21 at 11:39
Yes. I use same certificate loader elswhere with success. No error here. I did also check every command result with `(char*)ERR_error_string((int)ERR_get_error(), NULL);` getting no errors. `EVP_PKEY_CTX_ctrl_str` returns 1. — Krzysztof Derecki, Apr 08 '21 at 11:59
GOT IT! Problem was with my poor knowledge of C, i passed `EVP_CIPHER` pointer to `CMS_encrypt` but wrong way. After I put there raw `CMS_encrypt(NULL, dataBIOToEncrypt, EVP_aes_256_cbc(), flags);` it worked... thanks again for your time! Much appreciate — Krzysztof Derecki, Apr 08 '21 at 12:15

How to programmatically create encrypted PKCS#7 using ECC certificate and OpenSSL in C?

1 Answers1