error:FFFFFFFFFFFFFFFF:lib(255):func(4095):reason(4095) during RSA decyption

Question

I'm trying to create a hybrid cryptography tool in C++ with Qt Gui. (The data will be encrypted with AES 256-CBC, the AES Key RSA encrypted and saved then.) But the RSA part of this tool doesn't work. I wrote the sourcecode several times but I always get the same error on decrypt.

error:FFFFFFFFFFFFFFFF:lib(255):func(4095):reason(4095)

I hope someone could help me get a working RSA encrypt + decrypt implementation.

You can see the sourcecode here or download a testing Qt Project from my dropbox..

Dropbox Download: https://db.tt/6HKsYRTa

Sourcecode 1. Implementation:

void MainWindow::rsaEncrypt()
{
    EVP_PKEY *pk = NULL;
    EVP_PKEY_CTX *ctx = NULL;

    QByteArray encrypted = QByteArray();

    //------------------------------------------------
    //--- READ PUBLIC KEY ----------------------------
    FILE *pkFile = fopen(ui->publicKeyPath->text().toStdString().c_str(), "r");
    if(pkFile == NULL) throw NULL;

    pk = PEM_read_PUBKEY(pkFile, NULL, NULL, NULL);
    if(pk == NULL) throw NULL;
    fclose(pkFile);
    //------------------------------------------------
    ctx = EVP_PKEY_CTX_new(pk, NULL);


    //------------------------------------------------
    //--- ENCRYPT DATA -------------------------------
    int err;

    err = EVP_PKEY_encrypt_init(ctx);
    if(err <= 0) throw NULL;

    err = EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PADDING);
    if(err <= 0) throw NULL;

    size_t outLen = 0;
    err = EVP_PKEY_encrypt(
                            ctx,
                            NULL,
                            &outLen,
                            (uchar*) ui->plainTextEdit->document()->toPlainText().toStdString().c_str(),
                            ui->plainTextEdit->document()->toPlainText().size()
    );
    if(err <= 0) throw NULL;
    encrypted.resize(outLen);

    err = EVP_PKEY_encrypt(
                            ctx,
                            (uchar*) encrypted.data(),
                            &outLen,
                            (uchar*) ui->plainTextEdit->document()->toPlainText().toStdString().c_str(),
                            ui->plainTextEdit->document()->toPlainText().size()
    );
    //------------------------------------------------
    EVP_PKEY_CTX_free(ctx);
    EVP_PKEY_free(pk);

    if(err > 0) ui->encryptedTextEdit->document()->setPlainText(QString(encrypted));
    else {
            QByteArray errStr = QByteArray();
            errStr.resize(256);
            ERR_load_ERR_strings();
            ERR_error_string(err, errStr.data());
            ui->encryptedTextEdit->document()->setPlainText( QString(errStr) );
    }
}

void MainWindow::rsaDecrypt()
{
    EVP_PKEY *pk = NULL;
    EVP_PKEY_CTX *ctx = NULL;

    QByteArray decrypted = QByteArray();

    //------------------------------------------------
    //--- READ PRIVATE KEY ---------------------------
    FILE *pkFile = fopen(ui->privateKeyPath->text().toStdString().c_str(), "r");
    if(pkFile == NULL) throw NULL;

    pk = PEM_read_PrivateKey(pkFile, NULL, NULL, NULL);
    if(pk == NULL) throw NULL;
    fclose(pkFile);
    //------------------------------------------------
    ctx = EVP_PKEY_CTX_new(pk, NULL);


    //------------------------------------------------
    //--- DECRYPT DATA -------------------------------
    int err;

    err = EVP_PKEY_decrypt_init(ctx);
    if(err <= 0) throw NULL;

    err = EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PADDING);
    if(err <= 0) throw NULL;

    size_t outLen = 0;
    err = EVP_PKEY_decrypt(
                            ctx,
                            NULL,
                            &outLen,
                            (uchar*) ui->encryptedTextEdit->document()->toPlainText().toStdString().c_str(),
                            ui->encryptedTextEdit->document()->toPlainText().size()
    );
    if(err <= 0) throw NULL;
    decrypted.resize(outLen);

    err = EVP_PKEY_decrypt(
                            ctx,
                            (uchar*) decrypted.data(),
                            &outLen,
                            (uchar*) ui->encryptedTextEdit->document()->toPlainText().toStdString().c_str(),
                            ui->encryptedTextEdit->document()->toPlainText().size()
    );
    //------------------------------------------------
    EVP_PKEY_CTX_free(ctx);
    EVP_PKEY_free(pk);

    if(err > 0) ui->decryptedTextEdit->document()->setPlainText(QString(decrypted));
    else {
            QByteArray errStr = QByteArray();
            errStr.resize(256);
            ERR_load_ERR_strings();
            ERR_error_string(err, errStr.data());
            ui->decryptedTextEdit->document()->setPlainText( QString(errStr) );
    }
}

Sourcecode 2. Implementation:

void MainWindow::rsaEncrypt()
{
    RSA *rsa = createRSAFromFile(ui->publicKeyPath->text().toStdString().c_str(), 1);

    QByteArray encrypted = QByteArray();
    encrypted.resize(2048);

    int err = RSA_public_encrypt(
                    ui->plainTextEdit->document()->toPlainText().size(),
                    (uchar*) ui->plainTextEdit->document()->toPlainText().toStdString().c_str(),
                    (uchar*) encrypted.data(),
                    rsa,
                    RSA_PADDING
    );

    RSA_free(rsa);
    if(err > 0) ui->encryptedTextEdit->document()->setPlainText( QString(encrypted) );
    else {
            QByteArray errStr = QByteArray();
            errStr.resize(256);
            ERR_load_ERR_strings();
            ERR_error_string(err, errStr.data());
            ui->encryptedTextEdit->document()->setPlainText( QString(errStr) );
    }
}

void MainWindow::rsaDecrypt()
{
    RSA *rsa = createRSAFromFile(ui->privateKeyPath->text().toStdString().c_str(), 0);

    QByteArray decrypted = QByteArray();
    decrypted.resize(2048);

    int err = RSA_private_decrypt(
                    ui->encryptedTextEdit->document()->toPlainText().size(),
                    (uchar*) ui->encryptedTextEdit->document()->toPlainText().toStdString().c_str(),
                    (uchar*) decrypted.data(),
                    rsa,
                    RSA_PADDING
    );


    RSA_free(rsa);
    if(err > 0) ui->decryptedTextEdit->document()->setPlainText( QString(decrypted) );
    else {
            QByteArray errStr = QByteArray();
            errStr.resize(256);
            ERR_load_ERR_strings();
            ERR_error_string(err, errStr.data());
            ui->decryptedTextEdit->document()->setPlainText( QString(errStr) );
    }
}

RSA *MainWindow::createRSAFromFile(const char *keyPath, int pub)
{
    FILE *keyFile = fopen(keyPath, "rb");
    if(keyFile==NULL)
    {
            return 0;
    }
    RSA *rsa = RSA_new();

    if(pub)
    {
            rsa = PEM_read_RSA_PUBKEY(keyFile, &rsa, NULL, NULL);
    }
    else
    {
            rsa = PEM_read_RSAPrivateKey(keyFile, &rsa, NULL, NULL);
    }
    fclose(keyFile);
    return rsa;
}

Includes and defines for both implementations:

#include "mainwindow.h"
#include "ui_mainwindow.h"

#include <QDebug>
#include <QByteArray>

#include <openssl/evp.h>
#include <openssl/rsa.h>
#include <openssl/pem.h>
#include <openssl/err.h>

#define RSA_PADDING RSA_PKCS1_OAEP_PADDING

[what's `RsaEncrypt` doing in `MainWindow`](http://blog.codinghorror.com/the-wrong-level-of-abstraction/)? — Dmitry Ledentsov, Sep 12 '14 at 13:14
@Theolodis: error is the return value of the functions EVP_PKEY_decrypt() and RSA_private_decrypt(). — H4ckHunt3r, Sep 12 '14 at 13:25
@DmitryLedentsov rsaEncrypt is in the MainWindow because this source was just a small version for testing it, I will write a class later (When the encryption and decryption is working). — H4ckHunt3r, Sep 12 '14 at 13:31
@H4ckHunt3r, I'd suggest GUI-independent unit tests for that purpose — Dmitry Ledentsov, Sep 12 '14 at 13:40
@DmitryLedentsov I tried it without gui too, but the result was the same error.. And I really have no idea how to fix this. :( — H4ckHunt3r, Sep 12 '14 at 14:08

score 0 · Answer 1 · answered Sep 14 '14 at 16:50

Try to use this Code for creating a RSA-object. It definetly works. You should read the .pem file before and then call this function:

RSA *CryptClassRSA::createRSA(unsigned char *key, int isPublic){
    RSA *rsa = NULL;
    BIO *keybio;
    keybio = BIO_new_mem_buf(key, -1);
    if (keybio==NULL){
        printf( "Failed to create key BIO");
        return NULL;
    }
    if(isPublic){
        rsa = PEM_read_bio_RSA_PUBKEY(keybio, &rsa,NULL, NULL);
    }
    else{
        rsa = PEM_read_bio_RSAPrivateKey(keybio, &rsa,NULL, NULL);
    }
    if(rsa == NULL){
        printf( "Failed to create RSA");
    }

    return rsa;
}

jww · Answer 2 · 2016-01-28T02:20:27.497

C++ RSA decrypt error:FFFFFFFFFFFFFFFF:lib(255):func(4095):reason(4095)

And

int err;
...
err = EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PADDING);
if(err <= 0) throw NULL;
...

Its not clear to me where the error is happening or how you are getting the errstr output of error:FFFFFFFFFFFFFFFF:lib(255):func(4095):reason(4095). I picked EVP_PKEY_CTX_set_rsa_padding as the snippet because I think the return value is -2 (and it has significance as explained below).

err is just a return code. To get the actual error, you need to call ERR_get_error. Maybe something like:

int rc;
unsigned long err;
...

rc = EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PADDING);
err = ERR_get_error();
if(rc <= 0)
{
    // err is valid
}

You should also visit the EVP_PKEY_CTX_set_rsa_padding and ERR_get_error man pages.

Often, OpenSSL will return 0 for success, so I'm not sure about err <= 0 in some places. Also, because your error is 0xffff...4095 (and not 4096), I think you are getting the -2 return value discussed in the man page:

EVP_PKEY_CTX_ctrl() and its macros return a positive value for success and 0 or a negative value for failure. In particular a return value of -2 indicates the operation is not supported by the public key algorithm.

Also note... If you gather your error and print it in hex:

rc = EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PADDING);
err = ERR_get_error();
if(rc <= 0)
{
    // err is valid
    std::cerr << std::hex << err << std::endl;
}

Then you can use the openssl errstr 0xNNNN to print it.

error:FFFFFFFFFFFFFFFF:lib(255):func(4095):reason(4095) during RSA decyption

2 Answers2