Using OpenSSL in FIPS mode with .NET

Question

I've build OpenSSL FIPS container version 2.0.1, then built OpenSSL 1.0.1c using that container (according to the instructions in User Guide for the OpenSSL FIPS Object Module v2.0):

SET FIPSDIR=C:\OpenSSL\FIPS

cd C:\OpenSSL\openssl-fips-2.0.1
ms\do_fips

cd C:\OpenSSL\openssl-1.0.1c
ms\do_nasm
nmake -f ms\ntdll.mak

I've got these imports in my .NET application (which is obviously using the libeay32.dll from above):

[DllImport("libeay32", CallingConvention = CallingConvention.Cdecl)]
public extern static void ERR_load_crypto_strings();

[DllImport("libeay32", CallingConvention = CallingConvention.Cdecl)]
public extern static int ERR_print_errors_fp(SafeHandle file);

[DllImport("libeay32", CallingConvention = CallingConvention.Cdecl)]
public extern static int FIPS_mode_set(int onoff);

The problem is, that when I call FIPS_mode_set(1), it is returning 0, which indicates that FIPS mode could not be enabled.

To try and find out why the problem is, I'm doing this:

ERR_load_crypto_strings();

using (FileStream fs = new FileStream...)
{
  ERR_print_errors_fp(fs.SafeFileHandle);
}

Which gives me this error message:

OPENSSL_Uplink(0FAF1000,08): no OPENSSL_Applink

I've tried searching for answers, but have come up empty - any suggestions would be much appreciated!

** UPDATE **

I tried building it using the same steps from a Visual Studio 2008 commandline instead of a Visual Studio 2010 commandline (which I had previously used) - and it worked!

But I have no clue why it won't work with Visual Studio 2010, and I don't want to keep Visual Studio 2008 hanging around forever - any idea how I can get it to build OK with Visual Studio 2010?

Answer 1

I have searched in vain for complete instructions on how to build an OpenSSL FIPS capable executable and only found frustration and dead ends. I have finally, through numerous trial and error runs, found the following to work without any build errors. I am posting my findings here in hopes that it will help a poor soul out there looking for a solution to this build problem (and hopefully save them the few hairs they still have left on their heads!).

Summary of how to generate FIPS 140-2 OpenSSL executable and libraries: (The build machine used is a Windows 7 Ultimate).

Do the following steps as 'Administrator'!!!

1- Install ActiveState Perl and make sure environment path includes the bin folder where Perl has been installed. (See http://www.activestate.com/activeperl/downloads)

2- Install Nasm and add path to %PATH% environment variable. (See http://sourceforge.net/projects/nasm)

3- Install Visual Studio 2008 Pro if not already installed (VS2010/2012/2013 do NOT seem to build correctly!!).

4- Unpack OpenSSL-FIPS-2.0.9 src distribution to a new folder (do NOT reuse a previous folder of the same!).

5- Unpack OpenSSL-1.0.2 src distribution to a new folder (again: do NOT reuse a previous folder of the same!).

6- Launch a Visual Studio 2008 DOS command window (Start->Programs->MS Visual Studio 2008->Visual Studio DOS Command WIndow).

7- In that DOS window: "Set FIPSDIR=C:\OpenSSL-FIPS-2.0.9" (or whichever folder name hosts the OpenSSL-FIPS-2.0.9 src distribution just unpacked in step 4 above. Also: Should specify absolute path for FIPSDIR!!!).

8- If building for x86 target, then make sure to: 'Set PROCESSOR_ARCHITECTURE=x86' from that DOS command window. (Else link failures!).

9- CD C:\OpenSSL-FIPS-2.0.9 (or equivalent) and type: 'ms\ms_fips'. Should see 'BUILD SUCCESS' at end of build.

10- CD C:\OpenSSL-1.0.2 and type the following in sequence at the DOS prompt:

    -Perl Configure VC-WIN32 fips --prefix=C:\Build-OpenSSL-FIPS209
        (C:\Build-OpenSSL-FIPS209 will contain the build results when invoking 'INSTALL'. See below.)

    -ms\do_nasm  (ms\do_ms is NOT acceptable for FIPS-140-2 certification! Will use MASM instead of NASM)
        (For DEBUG versions then use flag debug-VC-WIN32 instead, then open ms\nt.mak after executing ms\do_nasm and replace all occurrences of /Zi with /Z7. There should be three replacements. For 64-bit, use VC-WIN64A and ms\do_win64a. If ECP source distros were used in generating the FIPS library in the first step, then    corresponding "FIPS capable" OpenSSL must be built with the no­ec2m option.)

    -nmake -f ms\nt.mak  (for static libs. To build shared libs use: ntdll.mak)

    -nmake -f ms\nt.mak install

11- Done. Verify that the (--prefix) folder 'C:\Build-OpenSSL-FIPS209' now contains the libraries, executable file and header files.

Answer 2

In the openssl documentation there is mention of this error that suggests that you have not created and compiled/linked an appropriate applink.

Alternatively, I think possibly this error might occur if you've not linked the openssl library in with code with the correct runtime (e.g. /MD if you're compiling c/c++ wrapper code).

Answer 3

Try better using this: http://sourceforge.net/projects/openssl-net/ - it will save you lot's of questions.

OpenSSL is complex, hence the C# wrapper library is really good help. To use OpenSSL you need to use pinvoke, which this library definitely does.

Alternativelly you could use external utility stunnel http://www.stunnel.org/index.html, or use openssl direclty thru exec(), and this way you can gain more stability per operation, in case the .NET library doesnt perform well.

It would be really recommended to use WIndows Crypto API - http://msdn.microsoft.com/en-us/library/windows/desktop/aa388162(v=vs.85).aspx as it's standard based, it works the same way, thru C# and even MSIE, using the system certificate repository, which does also imports and exports.