How to verify digest using openssl library?

Question

I'm build a Licensing system in the following way. I generated a matching public and private keys:

openssl genrsa -out mykey.pem 1024
openssl rsa -in mykey.pem -des3 -out prv-key.pem
openssl rsa -in mykey.pem -pubout -out pub-key.pem

Now I took a license message which contains unique information about the user and license details. my product reads this message, verifies the information, and if everything goes as planned it reads the license policy and runs accordingly.

so I've taken that license file and digested it with a signature using my private key

openssl dgst -sha256 -sign prv-key.pem -out license.secret license

and now I'm sending both the license and the signed license files to the customer.

My Question is: How can i verify digest using the public key (pub-key.pem) on the Customer side from within a C program. I've looking libssl and openssl library but couldn't find a good example for digest verification

Answer 1

You'll want to use libopenssl-dev or the Windows equivalent to build a C program similar to this:

#include <stdio.h>
#include <stdlib.h>
#include <openssl/pem.h>
#include <openssl/rsa.h>
#include <openssl/sha.h>

// Buffer for file read operations. The buffer must be able to accomodate
// the RSA signature in whole (e.g. 4096-bit RSA key produces 512 byte signature)
#define BUFFER_SIZE 512
static unsigned char buffer[BUFFER_SIZE];

int main(int argc, char *argv[])
{
    if(argc != 4)
    {
        fprintf(stderr, "Usage: %s datafile signature_file public_key\n", argv[0]);
        return -1;
    }
    const char* filename = argv[1];
    const char* sigfile = argv[2];
    const char* pubkeyfile = argv[3];

    unsigned bytes = 0;

    // Calculate SHA256 digest for datafile
    FILE* datafile = fopen(filename , "rb");

    // Buffer to hold the calculated digest
    unsigned char digest[SHA256_DIGEST_LENGTH];
    SHA256_CTX ctx;
    SHA256_Init(&ctx);

    // Read data in chunks and feed it to OpenSSL SHA256
    while((bytes = fread(buffer, 1, BUFFER_SIZE, datafile)))
    {
        SHA256_Update(&ctx, buffer, bytes);
    }

    SHA256_Final(digest, &ctx);
    fclose(datafile);

    // Read signature from file
    FILE* sign = fopen (sigfile , "r");

    bytes = fread(buffer, 1, BUFFER_SIZE, sign);
    fclose(sign);

    // Verify that calculated digest and signature match
    FILE* pubkey = fopen(pubkeyfile, "r"); 

    // Read public key from file
    RSA* rsa_pubkey = PEM_read_RSA_PUBKEY(pubkey, NULL, NULL, NULL);

    // Decrypt signature (in buffer) and verify it matches
    // with the digest calculated from data file.
    int result = RSA_verify(NID_sha256, digest, SHA256_DIGEST_LENGTH,
                            buffer, bytes, rsa_pubkey);
    RSA_free(rsa_pubkey);
    fclose(pubkey);

    if(result == 1)
    {
        printf("Signature is valid\n");
        return 0;
    }
    else
    {
        printf("Signature is invalid\n");
        return 1;
    }
}

Also instead of passing in the public key as a file parameter you can also back it in easily by using xxd -i pub-key.pem to generate a nice C style header of the key which can be used by the program in place of pubkeyfile.

Example code from: https://pagefault.blog/2019/04/22/how-to-sign-and-verify-using-openssl/