Segmentation fault when trying to use EVP functions in OpenSSL

Question

I am trying to do public encryption with OpenSSL using RSA and its high-level envelope functions. However I cannot seem to get my head around them and I'm getting a segmentation fault. This condensed code from my project reproduces the problem:

#include <iostream>
#include <string>

#include <openssl/evp.h>
#include <openssl/pem.h>
#include <openssl/err.h>
#include <openssl/rand.h>

int main()
{
    EVP_CIPHER_CTX *rsaCtx;
    rsaCtx = new EVP_CIPHER_CTX;

    unsigned char *ek;
    size_t ekl;
    unsigned char *iv;
    size_t ivl;

    EVP_PKEY *keypair;
    keypair = NULL;

    EVP_CIPHER_CTX_init(rsaCtx);

    EVP_PKEY_CTX *ctx = EVP_PKEY_CTX_new_id(EVP_PKEY_RSA, NULL);
    EVP_PKEY_keygen_init(ctx);
    EVP_PKEY_CTX_set_rsa_keygen_bits(ctx, 2048);
    EVP_PKEY_keygen(ctx, &keypair);
    EVP_PKEY_CTX_free(ctx);

    ek = new unsigned char[EVP_PKEY_size(keypair)];
    iv = new unsigned char[EVP_MAX_IV_LENGTH];
    ivl = EVP_MAX_IV_LENGTH;

    std::string cipherText;
    std::string plainText = "A STRING";
    size_t encMsgLen = 0;
    size_t blockLen  = 0;

    EVP_SealInit(rsaCtx, EVP_aes_256_cbc(), &ek, (int*)ekl, iv, &keypair, 1);
    EVP_SealUpdate(rsaCtx, (unsigned char*)cipherText.c_str() + encMsgLen, (int*)&blockLen, (const unsigned char*)plainText.c_str(), (int)plainText.size() + 1);
    encMsgLen += blockLen;
    EVP_SealFinal(rsaCtx, (unsigned char*)cipherText.c_str() + encMsgLen, (int*)&blockLen);
    encMsgLen += blockLen;
    EVP_CIPHER_CTX_cleanup(rsaCtx);

    EVP_PKEY_free(keypair);
    delete[] ek;
    delete[] iv;
    delete rsaCtx;

    std::cout << cipherText;

    return 0;
}

I get a segmentation fault at the line EVP_SealInit(rsaCtx, EVP_aes_256_cbc(), &ek, (int*)ekl, iv, &keypair, 1);

What am I doing wrong?

softwariness · Accepted Answer · 2015-03-02T12:52:29.673

ekl is a size_t, and you are casting it to an (int*).

The docs for EVP_SealInit say:

The actual size of each encrypted secret key is written to the array ekl.

You're just passing one key, so passing the address of a single integer is sufficient, but you should be passing the address of that integer, e.g.:

EVP_SealInit(rsaCtx, EVP_aes_256_cbc(), &ek, reinterpret_cast<int*>(&ekl), iv, &keypair, 1);

Alternatively, just declare ekl as an int in the first place, and you can avoid the cast:

int ekl;
//...
EVP_SealInit(rsaCtx, EVP_aes_256_cbc(), &ek, &ekl, iv, &keypair, 1);

I'm surprised your compiler didn't warn you about using an uninitialized local variable.

UPDATE: There are some more problems with this code besides the segmentation fault.

You're passing the buffer from an empty std::string (cipherText) into EVP_SealUpdate and EVP_SealFinal. This isn't going to work in general, and may crash or corrupt memory if there isn't enough room in the buffer.

You should declare a buffer of suitable size for the output, perhaps as std::vector<unsigned char> cipherText(bufferSize);, and pass &cipherText[0] to get the pointer to the first element.

The data in cipherText isn't a human-readable string, it's binary data, and std::cout isn't suitable for displaying it.

Some more general notes:

Avoid C-style casts in C++, and where possible write the code so you don't need to cast at all (e.g. declare the integers as int rather than size_t if that's what the APIs are expecting).
Avoid explicit memory-management with new and delete where you can, e.g. by using std::vector<unsigned char> for the buffers.

I suggest having a look at the documentation for these functions again, or some other examples on the web for using them. Also, write some code which does the decryption step so you can test that the plain-text is round-tripping correctly.

Thank you for your answer, this removed the segmentation fault, but now I just get a blank string as the output. What could be wrong? — James Lovejoy, Mar 02 '15 at 11:21
I've updated with some information about what's wrong here. If this isn't enough to help you resolve the other problems, I suggest asking a new question about how to use the `EVP_Seal*` functions correctly, as this is really a separate question, and will also mean you're more likely to get an answer to that specific problem. — softwariness, Mar 02 '15 at 12:56
Is there a reliable way to calculate the required buffer sizes (one for the ciphertext during encryption and one for the plaintext during decryption)? I don't see how I would use `std::vector` for the API functions `EVP_SealFinal` etc. because I'm passing a raw buffer and therefore the functions cannot reallocate internally. — flowit, May 10 '21 at 19:45

score 0 · Answer 2 · answered Jul 03 '18 at 23:20

Using a too small buffer with EVP_SealX functions may create maddening consequences in seemingly unrelated parts of your code. This was my experience.

Putting in guards that ensure the cipher buffer is as big as the total plaintext input plus possible padding overhead will reduce risk.

Segmentation fault when trying to use EVP functions in OpenSSL

2 Answers2