To get started, you'll need the private key to decrypt the message. By "public key (64 bit encoded)", I'm guessing what you really have is a Base-64–encoded certificate, with a header line that says "----- BEGIN CERTIFICATE-----" and a footer that says "-----END CERTIFICATE-----".
If that's correct, you'll need to find the private key. This is sometimes stored in a PKCS #12 format file, with a ".p12" or ".pfx" extension. You'll need a password to access the private key if it is stored in such a file.
Alternatively, OpenSSL and other utilities use private key files that can be Base-64–encoded or binary. These have a variety of extensions, and may or may not be password-protected. If the file that you have has a header line of "-----BEGIN RSA PRIVATE KEY-----" or "-----BEGIN PRIVATE KEY-----", that is actually the private key.
Finally, Windows can store private keys in its internal key store.
When you clarify the location of the private key, please update your question.
If the private key is used on the Java side, it may be an attempt to perform a digital signature. While all of several Java providers I've tested produce correct results when (ab)used this way, if you are doing a signature, the Signature
class should be used. The C# code should use a signature object to "verify" the signature as well.
Encryption is performed with the private key. Since the public key is public, anyone can decrypt the message; i.e., the message is not confidential. Public keys are used by recipients to verify signed messages.