How to add a custom OpenSSL engine with OpenSSL and use from apache server?

Question

I have a custom-built OpenSSL engine. I'm trying to make changes to openssl.cnf to load this engine automatically. My ultimate goal is to use this engine for Apache mod-ssl.

Apache mod_ssl to use OpenSSL ENGINE on Ubuntu 14.04, address my issue and I tried to follow the suggested solution. I have installed OpenSSL 1.1.1c from source code with following configuration,

./config --prefix=/opt/openssl -DOPENSSL_LOAD_CONF --openssldir=/opt/openssl/ssl 

According to Where to copy custom openssl engine library in openssl 1.1.0, I added the following changes to openssl.cnf to load my engine automatically,

openssl_conf = openssl_def

[openssl_def]
engines = engine_section

[engine_section]
rsa-engine-new = rsa_section

[rsa_section]
engine_id = rsa-engine-new
#dynamic_path = /opt/openssl/lib/engines-1.1/rsa-engine-new.so  <-- Uncomment this line cause segmentation fault

After making the changes, running openssl engine shows the following,

root@ss:/opt/openssl/ssl# openssl engine 
rsa-engine-new
(rdrand) Intel RDRAND engine
(dynamic) Dynamic engine loading support
(rsa-engine-new) engine for testing 1
140496290879232:error:260AB089:engine routines:ENGINE_ctrl_cmd_string:invalid cmd name:crypto/engine/eng_ctrl.c:255:
140496290879232:error:260BC066:engine routines:int_engine_configure:engine configuration error:crypto/engine/eng_cnf.c:141:section=rsa_section, name=oid_section, value=new_oids
140496290879232:error:0E07606D:configuration file routines:module_run:module initialization error:crypto/conf/conf_mod.c:177:module=engines, value=engine_section, retcode=-1      

The output of openssl engine shows some error, but my engine loaded automatically and use as a default engine.

Then I install httpd-2.4.10 from the source code with the following configuration,

CFLAGS='-DSSL_EXPERIMENTAL_ENGINE -DSSL_ENGINE -DOPENSSL_LOAD_CONF' ./configure --prefix=/etc/apache2 --enable-ssl --with-ssl=/opt/openssl/ssl --with-pcre=/usr/local/pcre --enable-so

After the installation, I have uncommented Include conf/extra/httpd-ssl.conf from httpd.conf. I added the following changes to /etc/apache2/conf/extra/httpd-ssl.conf file,

SSLCryptoDevice rsa-engine-new  <-- line 31
#SSLCryptoDevice /opt/openssl/lib/engines-1.1/rsa-engine-new

When I try to restart the httpd server, I get he following error,

root@ss:/etc/apache2/bin# ./httpd -k restart
AH00526: Syntax error on line 31 of /etc/apache2/conf/extra/httpd-ssl.conf:
SSLCryptoDevice: Invalid argument; must be one of: 'builtin' (none), 'rdrand' (Intel RDRAND engine), 'dynamic' (Dynamic engine loading support)

So, my question is,

1. why openssl engine throws error when the engine is working? And how can I fix this?
2. How can I configure httpd-ssl.cnf to use mod-ssl?

Answer 1

I think both of your issues might be caused by version mismatch. You are building the engine from OpenSSL 1.1.1c sources, and you are trying to use it with another version of OpenSSL.

The interface between dynamic engines and OpenSSL core are likely version dependent, which manifests as the Segmentation fault error when you are loading the dynamic engine.

You should build the whole OpenSSL library from 1.1.1c sources, or find a version of sources for your engine that is the same version as your OpenSSL install.