I have some encrypted data, which is encrypted using rc2-64-cbc NO PADDING. I am able to decrypt fine.
The issue : Even if the encrypted content is encrypted with wrong key my decryption doesn't error out, instead it decrypts to some garbage value, as its rc2 and no padding I believe.
I tried from openssl Linux command prompt and my C/C++ program(using EVP_* API calls).
Is there any openssl option/way to detect this the wrong key used ? ( command line or EVP_* C/C++ API calls)
Unless some form of authentication was added to the encrypted data there is no way to know for certain. The best is to look for decrypted that "makes sense", the decrypted data will either be correct or appear as a sequence of bytes that can not be distinguished from random bytes.
Assuming no authentication was added to the message possible methods include:
There is some know correct bytes at a known location in the message, this is known as a crib, an example is in WWII German weather reports began with predictable text.
Make a test of the randomness.
If the data is text check for invalid characters such as 0x00 - 0x1f.
Think of other tests that apply to your data.
You can take the plain text, calculate the checksum and then encrypt both. Once you decrypt the cipher text (even with a wrong key), try calculating the checksum again for the deciphered text and I am sure it will fail.
