Crash during the function CryptEncrypt

Question

I'm writting little program that will generate RSA key pair, export\import keys and encrypt string. So, I have written this code:

void EncryptString(std::string data) 
{
int lenght = strlen(data.c_str());
DWORD temp = data.length() * sizeof(char);
DWORD possiblersa = 0;

unsigned char* buffer = new unsigned char[lenght];

std::copy(data.begin(), data.end(), buffer);

if (!CryptEncrypt(hKey, NULL, true, NULL, NULL, &possiblersa, NULL))
{
    printf("Error: %d\n", GetLastError());
    ExitThread(0);
}

if (!CryptEncrypt(hKey, NULL, true, NULL, buffer, &temp, possiblersa)) // Problem here
{
    printf("Error: %d\n", GetLastError());
    ExitThread(0);
}

DWORD dlen = 0;
if (!CryptBinaryToString(buffer, possiblersa, CRYPT_STRING_BASE64, NULL, &dlen))
{
    printf("Error: %d\n", GetLastError());
    ExitThread(0);
}

TCHAR* str = new TCHAR[dlen];

if (!CryptBinaryToString(buffer, possiblersa, CRYPT_STRING_BASE64, str, &dlen))
{
    printf("Error: %d\n", GetLastError());
    ExitThread(0);
}

for (DWORD i = 0; i < dlen; i++) 
{
    printf("%d\n", str);
}

delete[] buffer;
delete[] str;
}

CryptEncrypt ends with the crash. I don't know what should I do to fix this issue.

`int lenght = strlen(data.c_str());` - it's spelled "length" and you can use `data.size()` or `data.length` - as, in fact, you do on the next line. — Useless, Jul 16 '19 at 10:41
What happened when you examined your crash in the debugger? What crypto library are you using and what should the arguments be? What actually was the value of `possiblersa` in the crashing call? What is `hKey`? — Useless, Jul 16 '19 at 10:43
It is very confusing to use `NULL` as an argument that expects a `DWORD`. `NULL` really ought to be `nullptr` which is not a valid `DWORD` (but you are probably using a backwards compatibility mode where `NULL` is just zero). — Martin Bonner supports Monica, Jul 16 '19 at 10:43
@Useless He's using the standard crypto library on Windows: https://learn.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-cryptencrypt — Martin Bonner supports Monica, Jul 16 '19 at 10:44
Cheers! OK, so once you fix your calls not to abuse NULL (and assuming you can't use the new API since this one is deprecated), what were the _values_ of the arguments passed to the failing call? — Useless, Jul 16 '19 at 10:47
@veter0 No. That's the argument to `EncryptString`. What were the arguments to the call to `CryptEncrypt` which failed. — Martin Bonner supports Monica, Jul 16 '19 at 10:52
@MartinBonner, `CryptEncrypt(hKey, NULL, true, NULL, buffer, &temp, possiblersa)` — veter0, Jul 16 '19 at 10:55

score 1 · Accepted Answer · answered Jul 16 '19 at 10:51

1

CryptEncrypt(hKey, NULL, true, NULL, NULL, &possiblersa, NULL))

will store in possiblersa the amount of data that will be returned by encrypting zero bytes from the null pointer. You almost certainly need to pass in the actual data you want to encrypt (from data.c_str()).

CryptEncrypt(hKey, NULL, true, NULL, buffer, &temp, possiblersa)

This encrypts your plaintext and claims that the buffer you have provided is of length possiblersa. That is almost certainly not true: it is very likely to be the case that possiblersa is much larger than length.

You need to delay allocating the buffer (and copying the plaintext into it) until you have found how large the ciphertext buffer needs to be. (It will be at least as long as the plaintext, but it can be much longer.)

answered Jul 16 '19 at 10:51

Martin Bonner supports Monica

28,528
3
51
88

rsa-2048 can encrypt only 256 bytes. possiblersa = 256. temp = 14, what's wrong? – veter0 Jul 16 '19 at 10:54
1

as the answer says, the API told you it needed 256 bytes of output, and you told it to go ahead and write 256 bytes into your 14 byte buffer. – Useless Jul 16 '19 at 11:05
@veter0 : CryptEncrypt will be writing 256 bytes of ciphertext to `buffer`. You have allocated 14 bytes there. You have just overflowed the buffer (by quite a lot). – Martin Bonner supports Monica Jul 16 '19 at 11:05
ok, it's working, now I have another problem. decrypt output != original text – veter0 Jul 16 '19 at 11:40

Crash during the function CryptEncrypt

1 Answers1