verifying digest signed with private key using a C program

Question

I'll explain all the steps I've done so far and conclude with my question.

using OpenSSL 1.0.1e-fips 11 Feb 2013

Generating a private and public key

openssl genrsa -des3 -out private.pem 2048
openssl rsa -in private.pem -outform PEM -pubout -out public.pem

signing a message with digest and private key

openssl dgst -sha256 -sign private.pem -out message.secret message.txt

at this point I have a public key, a signed message ( with digest ) and the original message.

Part 1 - using CLI ( this one works )

Using the CLI I manage to verify the digest:

openssl dgst -sha256 -verify public.pem -signature message.secret message.txt

I get "Verified OK" as a return value.

Part 2 - Using C program

My program looks like this:

where:

msg is message.txt

signature is message.secret

pkey is the public key ( achieved using PEM_read_PUBKEY )

int verify_it(const byte* msg, size_t msg_len, byte* signature, EVP_PKEY* pkey) {
    EVP_MD_CTX      *ctx;
    size_t          sig_len;
    int             bool_ret;
    int             ret_val;

    ret_val = EXIT_SUCCESS;
    ctx = NULL;

    do
    {
        ctx = EVP_MD_CTX_create();    
        const EVP_MD* md = EVP_get_digestbyname( "SHA256" );

        EVP_DigestInit_ex( ctx, md, NULL );
        EVP_DigestVerifyInit( ctx, NULL, md, NULL, pkey );
        EVP_DigestVerifyUpdate(ctx, msg, msg_len);

        sig_len = 256;            
        if ( !EVP_DigestVerifyFinal( ctx, signature, sig_len ) 
                ERR_print_errors_fp( stdout )
        );

    } while(0);

    return ret_val;

}

this code returns a verification failure ( value of 0 ).

also the function ERR_print_errors_fp( stdout ) prints the following message:

140332412258152:error:04091077:lib(4):func(145):reason(119):rsa_sign.c:176

EDIT

thanks to @deniss I've managed to fix this one problem ( signature length was bad - probably a '\0' mid signature - so i just edit the length to be 256 )

but now I'm getting another problem -

140195987986280:error:04091068:lib(4):func(145):reason(104):rsa_sign.c:293

checking it with openssl errstr I've got

error:04091068:rsa routines:INT_RSA_VERIFY:bad signature

the way I acquire my signature is like this:

secret_fp = fopen( "message.secret", "rb" );
fseek( secret_fp, 0, SEEK_END );
file_len = ftell( secret_fp );
fseek( secret_fp, 0, SEEK_SET );

signature = malloc( file_len );
fread( signature, file_len, 1, secret_fp );

Curious: why do you have `bool_ret` declared `int`, not `bool`? — too honest for this site, Aug 30 '15 at 14:29
Check if you are reading `message.txt` correctly. Generate signature with `echo "test"|openssl dgst -sha256 -sign private.pem -out message.secret` and hardcode `msg` as `"test"` and `msg_len` as `4`. — , Sep 01 '15 at 13:18
when reading the message.txt ( = "test" ) it also reads the \n in the end. should i drop it when i hardcode it? anyway either of those options still give me the same problem - bad signature — Itay Sela, Sep 01 '15 at 13:27
should i maybe use the command `openssl dgst -sha256 -hex -sign private.pem -out message.secret message.txt` to make the message.secret file written in hexa? — Itay Sela, Sep 01 '15 at 14:39
Your code works for me. linux host, openssl version: `OpenSSL 1.0.1f 6 Jan 2014`, libssl version: `0x1000106F`. [My code and PEMs](https://www.dropbox.com/s/7v9pbzffg6yuv34/openssl_verify.zip?dl=0), private key password is 1234. test.secret is generated with `openssl dgst -sha256 -sign private.pem -out test.secret main.cpp` — , Sep 02 '15 at 00:16
creating the files works fine. it's the verification via C program that fails — Itay Sela, Sep 02 '15 at 05:54
I've found my problem - the msg_len sent wasn't accurate and therefore it changed the whole digest. you couln't have found it since it not in the post :\ — Itay Sela, Sep 02 '15 at 13:37

score 5 · Answer 1 · answered Sep 01 '15 at 12:31

You can always decode openssl error codes into meaningful messages with

openssl errstr <error-code>

Code 04091077 stands for error:04091077:rsa routines:INT_RSA_VERIFY:wrong signature length.

The most probable explanation:

message.secret is binary file
it has zero byte somewhere in the middle
strlen trims signature on this byte

verifying digest signed with private key using a C program

1 Answers1