Botan failed to decrypt data using RSA key pair

Question

I'm making a program that has a client and server, and to send data from the client to the server it uses encryption. The data is encrypted with a key and the key gets encrypted with the server's public RSA key, but the server fails to decrypt the key giving me the message Invalid public key ciphertext, cannot decrypt but sometimes it does work. The RSA keys are saved as plain text and sent to the client on request (public key only). The private key is used only by the server. To encrypt the key the client does the following:

Botan::AutoSeeded_RNG rng;
Botan::DataSource_Memory DSMPublicServer(serverPublicKey); // serverPublicKey = key from server
Botan::X509_PublicKey *X509Key_publicServer = Botan::X509::load_key(DSMPublicServer); // Load the key
std::unique_ptr <Botan::Public_Key> publicKeyServer(X509Key_publicServer); // The key used to encrypt
Botan::PK_Encryptor_EME encKey(*publicKeyServer, rng, "EME-PKCS1-v1_5");
std::vector <uint8_t> encKey_t = encKey.encrypt(key, rng);

And the server tries to decrypt it as follows:

Botan::AutoSeeded_RNG rngTest;
Botan::DataSource_Memory DSMPrivate(this->myKeyString); // myKeyString = server private key
Botan::PKCS8_PrivateKey *PKCS8Key_Private = Botan::PKCS8::load_key(DSMPrivate, rngTest) // Load the key
std::unique_ptr <Botan::Private_Key> privateKey(PKCS8Key_Private);
Botan::PK_Decryptor_EME dec(*privateKey, rngTest, "EME-PKCS1-v1_5"); // Decryptor
std::vector <uint8_t> dec_t = Botan::unlock(dec.decrypt(this->key)); // This throws errors
this->key.clear(); // The key used on the rest of the data
std::copy(dec_t.begin(), dec_t.end(), std::back_inserter(this->key)); // Put the decrypted key back

If I test this on the server with some data it works fine, but it seems that either data is lost during transmission, but it uses TCP so that should not be it or that the encryption does not run properly every time since it does work sometimes. Is there a way to validate the encrypted data so I can test it before sending or how could I fix the issue?

Answer 1

Ok, so after a lot of different attempts I've managed to get it to work. The thing I've changed is how the server gets the data, at first this was in a long message after the function request, but now the server requests it in parts from the client. The small packets seem to work fine and I don't have any more problems with the encryption. Why it did not work with the original code I don't know but from all the tests I've done, it could be in several places, on the client when encrypting or building the message, on the server when disassembling the message.
--Edit--
Since more info was requested here it is.
At first, the client sends data to the server in a long string with encrypted data i.e. somefilter|function|data where the filter is used for ZMQ, the function is a function call to the server, and data contained plain text, text encrypted using the key of the client and the key encrypted using the RSA public key of the server. The server then disassembled the message using the lengths of those parts which were also in the message, but this did not work properly. I have not been able to figure out where it goes wrong exactly. Instead, I overhauled the server and client to send small parts, the client would send a function request to the server and one part of data to distinguish the client, after this the server would send messages for the other info that was previously part of the long message string, so now all the data comes in its own small packed instead of a large message. This solution, however, was only possible because I can change the server and client, if this was not possible the solution would not work. Also, this is not an answer to the question I asked but the issue with Botan failing to decrypt has been resolved ever since I overhauled the server and client, so I'm not sure as to what was going on with that.