wc_RsaSSL_Verify returns BAD_FUNC_ARG, and I can't tell why

Question

I am trying to RSA public key decrypt a signed file using wolfcrypt - yes, I may or may not be abusing the "sign/verify" power of RSA to encrypt a separate AES key using the private key and decrypt using the public key.

Unfortunately, I am stuck at wc_RsaSSL_Verify() - for the life of me I can't figure out why it is returning BAD_FUNC_ARG - I figured an error like that should be immediately visible to somebody else so I'm deciding to call upon the collective powers of StackOverflow.

As far as I can tell, I'm giving the function what it's asking for - an input buffer, an output buffer, the size of each, and a pointer to the RsaKey struct. Here is a code snippet from the function in question:

bool VerifyWorker::GetAESKey()
{
  bool result = true;
  uint8_t en_aes_file_buff[VerifyWorkerLocal::RSA_KEY_SIZE];
  uint8_t de_aes_file_buff[VerifyWorkerLocal::RSA_KEY_SIZE];

  uint8_t* aes_iv_ptr = NULL;

  // keyfile filestream
  std::fstream aes_file;

  // rsa_key must be initialized
  if(rsa_key == NULL)
  {
     result = false;
  }

  // Open the key file and read it into a local buffer, then decrypt it and use it to initialize the
  // aes struct
  if(result)
  {
     aes_file.open(this->aes_key_file, std::ios_base::in | std::ios_base::binary);

     if(aes_file.fail())
     {
        // Unable to open file - perror?
        perror("GetAESKey");
        result = false;
     }
     else
     {
        aes_file.read(reinterpret_cast<char*>(en_aes_file_buff), VerifyWorkerLocal::RSA_KEY_SIZE + 1);
        if(!aes_file.eof())
        {
           // we didn't have enough space to read the whole signature!
           std::cerr << "aes_file read failed! " << aes_file.rdstate() << std::endl;
           result = false;
        }
     }
  }

  // "Unsign" the aes key file with RSA verify, and load the aes struct with the result
  if(result)
  {
     int wc_ret = 0;
     wc_ret = wc_RsaSSL_Verify(const_cast<const byte*>(en_aes_file_buff),
                               VerifyWorkerLocal::RSA_KEY_SIZE, reinterpret_cast<byte*>(&de_aes_file_buff),
                               VerifyWorkerLocal::RSA_KEY_SIZE, rsa_key);

The rsa_key is a private member initialized (successfully, using wc_PublicKeyDecode()) in a separate function with a public key DER file. I generated both the public and private key using OpenSSL - which should properly pad my AES key and iv file using PKCS#1 v1.5 b default.

I should also mention that I am using wolfssl version 3.9.8. Thanks!

`const_cast(en_aes_file_buff)` what `reinterpret_cast(&de_aes_file_buff)` whaaaat — Lightness Races in Orbit, Feb 14 '18 at 01:25
Except for pointing out those oddities, I think realistically we would need a [MCVE] to debug this (as normal) — Lightness Races in Orbit, Feb 14 '18 at 01:26

score 1 · Accepted Answer · answered Feb 14 '18 at 16:56

The issue, I found, was that the file that I had signed with my RSA key was not signed correctly. When I signed the file using OpenSSL, my cli invocation was

openssl rsautl -in keyfile -out keyfile -inkey private.pem -sign

Apparently, openssl does not like you to specify the same file for -in and -out. When I changed it to something like

openssl rsautl -in keyfile -out keyfile_signed -inkey private.pem -sign

I was actually able to verify the file using wc_RsaSSL_Verify.

So, like most stupid late-night, last hour software problems, I was looking in the wrong place entirely. I was a bit thrown off by the BAD_FUNC_ARG being returned and thought that it had to do explicitly with the format of the function arguments, not necessarily their content. Hopefully this answer is useful for somebody else, too.

Kaleb · Answer 2 · 2018-02-14T19:05:09.853

It sounds like you are trying to use RSA_Sign to perform an "Encrypt" of an AES key. Then I assume you are sending to a remote partner or computer who will then run an RSA_Verify operation to decrypt the AES key do I understand the scenario correctly?

If so I apologize it did not show up if you searched on how to do this initially but we actually have an example of doing exactly that here:

https://github.com/wolfSSL/wolfssl-examples/tree/master/signature/encryption-through-signing

That example includes two separate applications. The first app, "rsa-private-encrypt-app.c", will sign (encrypt) the "fake Aes Key" and output the result to a file. The second app, "rsa-public-decrypt-app.c", then opens the file that was output and does a verify (decrypt) on the data contained in the file to recover the original "fake Aes Key".

I may or may not be abusing the "sign/verify" power of RSA to encrypt a separate AES key using the private key and decrypt using the public key.

No not at all, that is a valid use of RSA sign/verify ASSUMING you are working with fixed-length inputs such as an AES key.

That's why we created the example! We actually had a user ask a very similar question on our forums awhile back which led to us making the example.

One thing to make note of though on the issues you encountered with openssl and wolfssl is actually talked about in the README:

https://github.com/wolfSSL/wolfssl-examples/blob/master/signature/encryption-through-signing/README.md

... Keep in mind this is not a TRUE RSA ENCRYPT and will likely not inter-op with other libraries that offer a RSA_PRIVATE_ENCRYPT type API.

This is a true SIGN operation.

If you have any other questions feel free to post them here (and add the wolfssl tag of course) or you can also send us an email anytime at support@wolfssl.com

Disclaimer: I work for wolfSSL Inc.

Thanks! This is will be very useful going forward. I would vote up but I don't have the required rep — zkabitz, Feb 14 '18 at 19:31
Can I ask - I found that I got BAD_FUNC_ARG just because of the contents of the buffer I was trying to RSA verify. I have a rudimentary understanding of RSA encryption but I was just wondering - under what circumstances would an input data buffer cause the function to return BAD_FUNC_ARG? — zkabitz, Feb 15 '18 at 22:32
@zkabitz, As our library is written in C we are very strict about variable values and we often require that the length of a buffer be passed along with the buffer itself (to guard against buffer overflows and smashing the stack bugs). You might pass in a perfectly legitimate buffer but if the length you send for that buffer is zero we will return a BAD_FUNC_ARG, also if for some reason we are asking for a pointer and the pointer is NULL (has not been initialized or malloced) then we will return a BAD_FUNC_ARG. Might I ask which API you are calling when this gets returned? — Kaleb, Feb 16 '18 at 06:14
see my answer to this question. It seems like it was just the encrypted data that I passed in to wc_RsaSSL_Verify was causing it, considering that a change to that data (not a change to other parameters) caused it to go away. Is it possible that a bad encrypted data set, when decrypted, could cause a PKCS unpadding error, since the decrypted data certainly does not contain valid padding? — zkabitz, Feb 16 '18 at 16:31
Oh yes, absolutely. Glad you were able to resolve it! Cheers - Kaleb — Kaleb, Feb 16 '18 at 17:30

wc_RsaSSL_Verify returns BAD_FUNC_ARG, and I can't tell why

2 Answers2