2

I've got a very simple encryption/decryption program that works fine without FIPS support enabled, but fails when it is:

    #include <openssl/conf.h>
    #include <openssl/evp.h>
    #include <openssl/err.h>
    #include <string.h>

    void handleErrors(void)
    {
        ERR_print_errors_fp(stderr);
        abort();
    }

    int encrypt(unsigned char *plaintext, int plaintext_len, unsigned char *key,
    unsigned char *iv, unsigned char *ciphertext)
{
  EVP_CIPHER_CTX *ctx;

  int len;

  int ciphertext_len;

  /* Create and initialise the context */
  if(!(ctx = EVP_CIPHER_CTX_new())) handleErrors();

  /* Initialise the encryption operation. IMPORTANT - ensure you use a key
   * and IV size appropriate for your cipher
   * In this example we are using 256 bit AES (i.e. a 256 bit key). The
   * IV size for *most* modes is the same as the block size. For AES this
   * is 128 bits */
  if(1 != EVP_EncryptInit_ex(ctx, EVP_aes_256_cbc(), NULL, key, iv))
    handleErrors();

  /* Provide the message to be encrypted, and obtain the encrypted output.
   * EVP_EncryptUpdate can be called multiple times if necessary
   */
  if(1 != EVP_EncryptUpdate(ctx, ciphertext, &len, plaintext, plaintext_len))
    handleErrors();
  ciphertext_len = len;

  /* Finalise the encryption. Further ciphertext bytes may be written at
   * this stage.
   */
  if(1 != EVP_EncryptFinal_ex(ctx, ciphertext + len, &len)) handleErrors();
  ciphertext_len += len;

  /* Clean up */
  EVP_CIPHER_CTX_free(ctx);

  return ciphertext_len;
}

int decrypt(unsigned char *ciphertext, int ciphertext_len, unsigned char *key,
  unsigned char *iv, unsigned char *plaintext)
{
  EVP_CIPHER_CTX *ctx;

  int len;

  int plaintext_len;

  /* Create and initialise the context */
  if(!(ctx = EVP_CIPHER_CTX_new())) handleErrors();

  /* Initialise the decryption operation. IMPORTANT - ensure you use a key
   * and IV size appropriate for your cipher
   * In this example we are using 256 bit AES (i.e. a 256 bit key). The
   * IV size for *most* modes is the same as the block size. For AES this
   * is 128 bits */
  if(1 != EVP_DecryptInit_ex(ctx, EVP_aes_256_cbc(), NULL, key, iv))
    handleErrors();

  /* Provide the message to be decrypted, and obtain the plaintext output.
   * EVP_DecryptUpdate can be called multiple times if necessary
   */
  if(1 != EVP_DecryptUpdate(ctx, plaintext, &len, ciphertext, ciphertext_len))
    handleErrors();
  plaintext_len = len;

  /* Finalise the decryption. Further plaintext bytes may be written at
   * this stage.
   */
  if(1 != EVP_DecryptFinal_ex(ctx, plaintext + len, &len)) handleErrors();
  plaintext_len += len;

  /* Clean up */
  EVP_CIPHER_CTX_free(ctx);

  return plaintext_len;
}

int main (void)
{
  /* Force FIPS initialization */
  FIPS_mode_set(1);
  /* Set up the key and iv. Do I need to say to not hard code these in a
   * real application? :-)
   */

  /* A 256 bit key */
  unsigned char *key = (unsigned char *)"01234567890123456789012345678901";

  /* A 128 bit IV */
  unsigned char *iv = (unsigned char *)"01234567890123456";

  /* Message to be encrypted */
  unsigned char *plaintext =
                (unsigned char *)"The quick brown fox jumps over the lazy dog";

  /* Buffer for ciphertext. Ensure the buffer is long enough for the
   * ciphertext which may be longer than the plaintext, dependant on the
   * algorithm and mode
   */
  unsigned char ciphertext[128];

  /* Buffer for the decrypted text */
  unsigned char decryptedtext[128];

  int decryptedtext_len, ciphertext_len;

  /* Initialise the library */
  ERR_load_crypto_strings();
  OpenSSL_add_all_algorithms();
  OPENSSL_config(NULL);

  /* Encrypt the plaintext */
  ciphertext_len = encrypt (plaintext, strlen ((char *)plaintext), key, iv,
                            ciphertext);

  /* Do something useful with the ciphertext here */
  printf("Ciphertext is:\n");
  BIO_dump_fp (stdout, (const char *)ciphertext, ciphertext_len);

  /* Decrypt the ciphertext */
  decryptedtext_len = decrypt(ciphertext, ciphertext_len, key, iv,
    decryptedtext);

  /* Add a NULL terminator. We are expecting printable text */
  decryptedtext[decryptedtext_len] = '\0';

  /* Show the decrypted text */
  printf("Decrypted text is:\n");
  printf("%s\n", decryptedtext);

  /* Clean up */
  EVP_cleanup();
  ERR_free_strings();

  return 0;
}

As you can see, just the demo code with FIPS enabled. Without FIPS, my output is:

Ciphertext is:
0000 - e0 6f 63 a7 11 e8 b7 aa-9f 94 40 10 7d 46 80 a1   .oc.......@.}F..
0010 - 17 99 43 80 ea 31 d2 a2-99 b9 53 02 d4 39 b9 70   ..C..1....S..9.p
0020 - 2c 8e 65 a9 92 36 ec 92-07 04 91 5c f1 a9 8a 44   ,.e..6.....\...D
Decrypted text is:
The quick brown fox jumps over the lazy dog

With FIPS, compilation goes fine, but generates the following when run:

139686960322208:error:2D0A0086:FIPS routines:FIPS_cipher:selftest failed:fips_enc.c:336:
139686960322208:error:2D0A0086:FIPS routines:FIPS_cipher:selftest failed:fips_enc.c:336:

I've tried both as a C project, and as a C++ project, pointing the CC env variable at both the fipsld script, and the modified fipsld++ script as appropriate. My FIPSLD_CC variable points to gcc as noted in the FIPS documentation.

What am I missing here?

Matthew Heimlich
  • 343
  • 2
  • 13
  • What version of _OpenSSL_ are you using? Is it compiled as a static or dynamic lib? Does it work in _FIPS_ mode? (I am talking about _openssl.exe_ here). – CristiFati Oct 07 '16 at 16:16
  • OpenSSL version is 1.0.2j, FIPS module version is 2.0.13. Libraries are static. Not sure how to run OpenSSL in FIPS mode (on Linux here, btw, so obviously no .exe, but same concept.) – Matthew Heimlich Oct 07 '16 at 16:46
  • I've done some further research, enabled FIPS compliancy for RHEL as a whole, and now OpenSSL fails to produce md5 (as expected) but properly produces sha1. Seems that FIPS support is working there. – Matthew Heimlich Oct 07 '16 at 17:11
  • You have to check the return value from `FIPS_mode_set(1)` and the other operations. Also, it appears the binary has a good signature. The error code is different for a failed integrity check. I believe error code 0x2D0A0086 is `FIPS_R_FIPS_SELFTEST_FAILED`, and the explanation from th user guide is *"An algorithm known answer tests has failed"*. A failed integrity check returns `FIPS_R_FINGERPRINT_DOES_NOT_MATCH`. Finally, you can probably reduce the problem to an empty `main` that only calls `FIPS_mode_set(1)`. Also see Section D.3 Error Codes in the FIPS User Guide. – jww Oct 07 '16 at 18:10
  • So, simply setting FIPS mode in a blank project returns no errors. Unfortunately, the error code section in the user guide is next to useless. Not much there in the way of figuring out why or how it's failed. Running FIPS_mode_set(1) and then immediately running printf("%d", FIPS_mode()) returns a 0. Seems incorrect to me. Any ideas? – Matthew Heimlich Oct 07 '16 at 18:53
  • Sorry for the _.exe_ part. So, `FIPS_mode_set(1)` returns 1, and then `FIPS_mode()` returns 0? Any error on the stack? [`ERR_peek_last_error`](https://www.openssl.org/docs/manmaster/crypto/ERR_get_error.html). I'd go for the dynamic _OpenSSL + FIPS_ version because (besides the obvious reasons) the _FIPS_ build would only happen once (when building _libcrypto.so_), and any of its clients would build normally (unless there's an impediment distributing the lib). P.S.: When running _openssl_ executable in _FIPS_ mode, simply `export OPENSSL_FIPS=1` env var (unless _FIPS_ set at machine level). – CristiFati Oct 07 '16 at 19:35
  • @CristiFati Doing an error peek directly after setting the mode returns error code 2d06b06f, for which errstr returns `error:2D06B06F:FIPS routines:DSA_BUILTIN_PARAMGEN2:fingerprint does not match nonpic relocated` Any tips on compiling dynamically? The docs for compiling as is already felt lacking. I'm not so concerned with distribution at the moment, just working on something as proof of concept for work. – Matthew Heimlich Oct 07 '16 at 20:07
  • @MatthewHeimlich - `FIPS_mode_set` returns what you set it to. If you set it to 1, then it returns 1 and others are failures. If you set it to 2 (NSA Suite B algorithms), then anything other than 2 is a failure. Now that you are checking return values and error codes, 0x2D06B06F sounds a little more like what I am used to seeing. You said `FIPS_mode_set` works for EXE, which means your EXE had the signature embedded. It sounds like 0x2D06B06F is related to the shared object. – jww Oct 08 '16 at 04:30
  • @MatthewHeimlich - I'd look at two things. (1) Does `ldd` show you loading the share object you built? Or is it loading the one from Fedora or RH? (2) Does your shared object have a signature on it? I believe the symbol of interest is `unsigned char FIPS_signature[20]` (it may be 32 bytes in size now). Is it all 0's due to static initialization, or is it populated? Also keep in mind: be careful of debugging with software breakpoints. They break the integrity check if they are placed on the sequestered code or data. – jww Oct 08 '16 at 04:30
  • You should probably delete this question and ask a new question. I'm having trouble following the various comments that have information that should be in the question. Take what you have learned, craft a new minimal example (empty main?), state exactly the configuration which fails (EXE vs SO), state exactly where and how it fails (API call and error code), and craft a new question. – jww Oct 08 '16 at 05:28
  • [Here](https://www.openssl.org/docs/fips/UserGuide-2.0.pdf) is the _OpenSSL_ (+ _FIPS_ ) build process (chapters 4 and 5), probably this is the location that you read the build instructions from. Seems that the signature computed at linktime (and embedded into the executable) is not the same with the one computed at runtime. But what's strange is that (after a quick sourcecode browse) `OPENSSL_NONPIC_relocated` seems related to _Win_. – CristiFati Oct 11 '16 at 10:21

0 Answers0