rsa encryption in windows crypto and decryption in openssl

Question

I have a problem in exchanging public key between "OpenSSL" and "Windows CryptoAPI". The public key is exported from OpenSSL in pem format.My program is written in c++. I get the public key and load it by "CryptoAPI". After loading the public key, I encrypt some data and send them to the other application. The other application can not decrypt the received data by own private key. please help me to find the solution.

Source code:

_ServerContextHandle = NULL;
_EncryptionKeyHandle = NULL;

void Initialize(char* inPublicKeyByPemFormat)
{
    HCRYPTPROV serverContextHandle;

    bool result = CryptAcquireContextW(&serverContextHandle, nullptr, nullptr, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT) == TRUE;

    if (result)
    {
        _ServerContextHandle = serverContextHandle;

        byte derPublicKey[2048];
        DWORD derPublicKeyLength = 2048;

        result = CryptStringToBinaryA(inPublicKeyByPemFormat, 0, CRYPT_STRING_BASE64HEADER, derPublicKey, &derPublicKeyLength, nullptr, nullptr) == TRUE;

        CERT_PUBLIC_KEY_INFO* publicKeyInfo = nullptr;
        DWORD publicKeyInfoLength;

        if(result)
        {
            result = CryptDecodeObjectEx(X509_ASN_ENCODING | PKCS_7_ASN_ENCODING, X509_PUBLIC_KEY_INFO, derPublicKey, derPublicKeyLength, CRYPT_ENCODE_ALLOC_FLAG, nullptr, &publicKeyInfo, &publicKeyInfoLength) == TRUE;
        }

        HCRYPTKEY encryptionKeyHandle;

        if(result)
        {
            result = CryptImportPublicKeyInfo(_ServerContextHandle, X509_ASN_ENCODING, publicKeyInfo, &encryptionKeyHandle) == TRUE;
        }

        LocalFree(publicKeyInfo);

        if (result)
        {
            _EncryptionKeyHandle = encryptionKeyHandle;
        }
    }
}


byte* EncryptData(byte* inData, DWORD inDataLength, DWORD* outLength) const
{
    byte* result = nullptr;

    *outLength = 0;

    DWORD length = inDataLength;

    result = CloneByteArray(inData, inDataLength);

    if (!CryptEncrypt(_EncryptionKeyHandle, NULL, TRUE, 0, result, &length, length))
    {
        delet result;

        result = new byte[length];

        CopyByteArray(inData, result, inDataLength);

        *outLength = inDataLength;

        if (!CryptEncrypt(_EncryptionKeyHandle, NULL, TRUE, 0, result, outLength, length))
        {
            delete result;

            result = nullptr;

            *outLength = 0;
        }
    }
    else
    {
        *outLength = length;
    }

    return result;
}

Without the necessary information we cannot debug, even if we would like to debug your code (please debug this is not a question, it's a request for help). — Maarten Bodewes, Jun 18 '17 at 18:20
In your second function inside its first if statement you are deleting the result value but you did not spell it correctly. — Francis Cugler, Jun 18 '17 at 18:23
You should state where the failure is in the code above. Is it `CryptDecodeObjectEx` or `CryptImportPublicKeyInfo` or something else? You should validate the public key after loading it. It may mean you need to load a test key pair to ensure correctness. OpenSSL is big endian, and CAPI is little endian. You should probably start with [openssl capi rsa endian site:stackoverflow.com](https://www.google.com/search?q=openssl+capi+rsa+endian+site%3Astackoverflow.com). And see [Load an X509 PEM file into Windows CryptoApi](https://stackoverflow.com/q/1231178/608639) in particular. — jww, Jun 18 '17 at 19:00

score 2 · Accepted Answer · answered Jun 18 '17 at 20:47

CryptEncrypt, for reasons that probably made sense to the author at the time, writes the bytes down backwards. Or, rather, it writes them in byte minor/little-endian order whereas almost every other cryptography library (including the Windows CNG BCryptEncrypt and NCryptEncrypt routines) writes them in byte major/big-endian order.

So you need to reverse data coming out of CryptEncrypt, and reverse it going in to CryptDecrypt.

For example, in .NET Core's RSACryptoServiceProvider.Encrypt it calls CapiHelper.EncryptKey which calls CryptEncrypt then Array.Reverse before returning.

The CryptEncrypt documentation has as the last sentence in the Remarks section

The ciphertext is returned in little-endian format.

rsa encryption in windows crypto and decryption in openssl

1 Answers1