error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding openssl C++

Question

I am trying to verify a signature using OpenSSL in C++. The code is below:

bool verifySignature(QString updateInfo_file_signature, QString uInfo_file_hash) {
    RSA *rsa = NULL;
    BIO *keybio;

    FILE *file = fopen("pubkey.pem", "rb");
    fseek(file, 0, SEEK_END);
    long fsize = ftell(file);
    fseek(file, 0, SEEK_SET);

    char *key = (char*)malloc(fsize + 1);
    fread(key, 1, fsize, file);
    fclose(file);

    keybio = BIO_new_mem_buf((void*)key, -1);
    if (keybio==NULL)
        return 0;

    rsa = PEM_read_bio_RSA_PUBKEY(keybio, &rsa,NULL, NULL);
    if(rsa == NULL)
        return 0;
    
    int rsa_size = RSA_size(rsa);

    // vars
    const char* sign_file_hash = uInfo_file_hash.toStdString().c_str();
    unsigned char* u_file_hash = (unsigned char*) uInfo_file_hash.toLocal8Bit().data();

    const char* sign_file_sig = updateInfo_file_signature.toStdString().c_str();
    unsigned char* u_file_sig = (unsigned char*) updateInfo_file_signature.toLocal8Bit().data();

    int hash_size = strlen(sign_file_hash);
    int sig_size = strlen(sign_file_sig);

    int res = RSA_verify(NID_sha1, u_file_hash, 16, u_file_sig, rsa_size, rsa);
    printError();

    return 0;
}

But I am getting this error:

error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding

Here is my public key:

Here is file Signature (updateInfo_file_signature::QString): saKBgdDIS/rsb7Uazr6zWMYsGLU8CYN6YaUZh5nyNjo7PCImCNtXBV+4TuFnKV6obz1rdqqUX+0Lwan8gquqQzYJFFQZFVexHSEyzxPZXYLmyFU35Gbko/iSGlkg8F/DVCSPsSttlhhQJjjHCbMB9i+DgzFMCDYVhd9lrtuEVDauXDtuEZi5MtEbyA1G3i5LT9H6Hr7XUTQN7QAnbHxCdtPc81FHO9+WEdu/lDdmT+rfWKO1REEeOVd/0Pf/pGTCVdnVsCA+S3UD310Ft13UB8KyQ5xN/KrncUFibaKzzKShR2/pXPHWWhkP5Ceku4cJOiV7YY9+ZUPMV7rfJq9KDw==

Here is computed File hash in sha256 (uInfo_file_hash::QString): 712b6ec279d490ede7454f34d1f6ffff

I have tried cat -v pubkey.pem but still it seems legit.

I've been struggling with it for days, please do some magic and help me out.

P.S: Please excuse any memory leaks or unused variables because this is a code snippet and they are taken care of later.

Welcome to StackOverflow. It seems there is something wrong with the public key. Please provide info about how it was generated. — f9c69e9781fa194211448473495534, Nov 19 '20 at 20:11
Thank you :) Public key is generated by OpenSSL using command line: `openssl dgst -sign privatekey.pem -keyform PEM -sha256 -out signaturefile.sign -binary package.deb` — why it doesnt work, Nov 20 '20 at 02:28
That does not look like the right command for generating a public key from a private key. See for example this [answer](https://stackoverflow.com/a/10271238/10992173) for how to obtain the public key. — f9c69e9781fa194211448473495534, Nov 21 '20 at 12:40
Ops, you are right. Actually I am working on my module, i.e., verification. and do not know how key was generated. Also I checked the key with `openssl asn1parse -in pubkey.pem` The key is **valid**. The output is as follows: `0:d=0 hl=4 l= 290 cons: SEQUENCE, 4:d=1 hl=2 l= 13 cons: SEQUENCE, 6:d=2 hl=2 l= 9 prim: OBJECT:rsaEncryption, 17:d=2 hl=2 l= 0 prim: NULL, 19:d=1 hl=4 l= 271 prim: BIT STRING` — why it doesnt work, Nov 22 '20 at 18:08
Weird, I cannot reproduce that output with the key in your question. `openssl asn1parse -in pubkey.pem` gives me: `139795141531008:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:crypto/asn1/asn1_lib.c:101`. — f9c69e9781fa194211448473495534, Nov 22 '20 at 18:41

score 1 · Answer 1 · answered Nov 21 '20 at 13:47

I think the problem is that you are storing a pointer to a temporary variable that has gone out of scope by the time you use it:

const char* sign_file_hash = uInfo_file_hash.toStdString().c_str();

The Qstring::toStdString() method return a std::string by value, but you are not storing the std::string anywhere. Instead, you get the pointer to its contents using .c_str(), but after this line has executed, the std::string no longer exist. The same happens in this line:

unsigned char* u_file_hash = (unsigned char*) uInfo_file_hash.toLocal8Bit().data();

The solution is to store the temporary in a variable first:

QByteArray u_file_hash = uInfo_file_hash.toLocal8Bit();

And apply .data() as late as possible:

int res = RSA_verify(NID_sha1, u_file_hash.data(), 16, u_file_sig.data(), rsa_size, rsa);

I also see that sign_file_hash, sign_file_sig, hash_size and sig_size are not actually used for anything.

error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding openssl C++

1 Answers1