Testing a custom openSSL engine from the command line

Question

Quick question: How can I verify the functionality of a custom openSSL engine I'm writing from the command line?

Right now I am following along with this great tutorial, and am successfully able to exercise the engine (which returns a digest value of all 2's) using my test program (source code here, test program located at test/wssha256engine_test.c).

brett@wintermute:~/openssl_ws/wssha256engine$ make test
make[1]: Entering directory 
/home/brett/openssl_ws/wssha256engine/test
make[1]: '../bin/test' is up to date.
make[1]: Leaving directory 
/home/brett/openssl_ws/wssha256engine/test
brett@wintermute:~/openssl_ws/wssha256engine$ bin/test
Engine Loaded...
Initializing wssha256 engine...
*TEST: Successfuly initialized engine [A test engine for the ws sha256 hardware encryption module, on the Xilinx ZYNQ7000]
    init result = 1
Digest NID=672 requested
SHA256 algorithm context initialized
*TEST: Digest INIT 1
SHA256 update
*TEST: Digest Update 1
SHA256 final: sizeof(EVP_MD)= 120
SHA256 cleanup
*TEST: Digest Final 1 Digest size:32
22222222222222222222222222222222

However, because reasons, I would like to also use the openssl command line interface to exercise the engine I just wrote, and have it compute the digest of a random string like in this other tutorial, just using sha256 and not md5.

But when I try and do this, the engine does not load and results in an error telling me that the NID of my digest doesn't exist, and just hashes the string with the standard algorithm instead:

brett@wintermute:~/openssl_ws/wssha256engine$ echo "Hello" | openssl dgst -engine /home/brett/Thesis/openssl_ws/wssha256engine/bin/libwssha256engine.so -sha256
ERROR: Digest is empty! (NID = 0)
engine "wssha256" set.
(stdin)= 66a045b452102c59d840ec097d59d9467e13a3f34f6494e539ffd32c1bb35f18

Why can't I use my engine on the command line, but I can create a C program to load it? And how can I use my engine to compute a digest from the command line?

Note: I am able to load the engine from the command line, just not use it.

If I follow your final question, the answer would be that your `shell` does not know anything about initializing the openssl environment that is loaded by the C program you write to load it. As far as I know there is no way to tell the openssl interface to use a separate engine either. If you want to make your engine available through a CLI interface, then you would need to implement a small shell (via `execv`, etc.) to pass commands to your engine, or provide an input routine for your engine you can redirect the wanted commands to. — David C. Rankin, May 15 '17 at 18:13
@DavidC.Rankin I don't think that is the case. A number of tutorials use the engine API the way I am intending — asmvolatile, May 17 '17 at 12:36
Now you are making more sense. Your C program was written pursuant to specific API. You are saying that the openssl CLI provides specific hooks for the API you used, but when you try with your C program you receive the `ERROR: Digest is empty!` as you try to initialize the engine. Have you complied with **all** requirements of the **Preparations** section of the page? Specifically the "*OpenSSL Engines are stored in /usr/lib/engines/*"? — David C. Rankin, May 17 '17 at 15:48

score 0 · Accepted Answer · answered May 22 '17 at 16:39

After some digging, I was able to figure out why the CLI invocation of my engine did not work, and fix the issue.

The problem was in my digest selector function (original broken code is commented out below). Originally, I returned a failure status from the digest selector function, rather than returning the number of digest IDs that the engine supports.

static int wssha256engine_digest_selector(ENGINE *e, const EVP_MD **digest, const int **nids, int nid)
{
    if (!digest)
  {
    *nids = wssha256_digest_ids;

    // THIS ORIGINAL CODE CAUSES THE ENGINE TO NEVER INITIALIZE FROM A CLI INVOCATION
    //printf("ERROR: Digest is empty! (NID = %d)\n",nid);
    //return FAIL;

    // THIS FIXES IT
    int retnids = sizeof(wssha256_digest_ids - 1) / sizeof(wssha256_digest_ids[0]);
    printf("wssha256engine: digest nids requested...returning [%d]\n",retnids);
    return retnids;
}

Reading more closely, we can find that OpenSSL calls the digest selector function in following (separate) ways:

with digest being NULL. In this case, *nids is expected to be assigned a zero-terminated array of NIDs and the call returns with the number of available NIDs. OpenSSL uses this to determine what digests are supported by this engine.
with digest being non-NULL. In this case, *digest is expected to be assigned the pointer to the EVP_MD structure corresponding to the NID given by nid. The call returns with 1 if the request NID was one supported by this engine, otherwise 0.

So even though my engine supported sha256, the CLI invocation didn't work because it must have not been initialized with a digest before the digest selector function is invoked (no clue as to the order of operations when using the CLI). However when I manually initialized it in my test program, all worked fine. Changing to a non-failure return value to support invocation 1. above fixed the problem.

Testing a custom openSSL engine from the command line

1 Answers1