RSA key generation error in RSA_generate_key_ex function call

Question

I'm trying to generate RSA keys using openssl. I've got error on RSA_generate_key_ex and have no idea why the error occures.

I combine ERR_get_error() and ERR_error_string() and I've got next message: error:2506906C:lib(37):func(105):reason(108). Also I found out that 108 error code means RSA_R_DATA_GREATER_THAN_MOD_LEN

I'm trying to generate RSA keys using C code below. For sake of brevity I reduce free calls and error output

RSA* generateRSA()
{
  BIGNUM *bne = BN_new();
  if (bne == NULL)
  {
    return NULL;
  }
  if (BN_set_word(bne, RSA_F4) != 1)
  {
    return NULL;
  }

  RSA *r = RSA_new();
  if (r == NULL)
  {
    return NULL;
  }
  // THERE I'VE GOT ERROR
  if (RSA_generate_key_ex(r, 2048, bne, NULL)!= 1)
  {
    // ERR_get_error() returns 2506906C
    // ERR_error_string() gives me RSA_R_DATA_GREATER_THAN_MOD_LEN
    return NULL;
  }

  return r;
}

The question is what does the error mean and how can I fix it?

Edit: I use OpenSSL 1.1.0e 16 Feb 2017. I use it as part of EDK II Project

Answer 1

I figured out that random generator need to be seeded (openssl versions 1.0.2 and 1.1.0 random generators must be explicitly seeded).

I check RAND_status(). It returned 0. So the solution is just add RAND_seed() before key generation:

const void* getSeedBuffer(int num);

RSA* generateRSA()
{
  RAND_seed(getSeedBuffer(1000), 1000); // don't forget to free memory
  BIGNUM *bne = BN_new();
  if (bne == NULL)
  {
    return NULL;
  }
  if (BN_set_word(bne, RSA_F4) != 1)
  {
    return NULL;
  }

  RSA *r = RSA_new();
  if (r == NULL)
  {
    return NULL;
  }
  if (RSA_generate_key_ex(r, 2048, bne, NULL)!= 1)
  {
    return NULL;
  }

  return r;
}