'IDX10618: Key unwrap failed using decryption Keys for JWE

Question

Our team is implementing a microservice architecture, and in this implementation we have a service providing JWE's (encrypted JWT) upon authentication. Our other services are responsible for verifying the validity of the JWE and ensuring the user can actually use the service.

I have created a brand new DotNet 6 Web API that uses the Microsoft.AspNetCore.Authentication.JwtBearer and Microsoft.IdentityModel.Tokens nuget packages. I put the [Authorize] attribute on the WeatherController and have set up the TokenValidationParameters with the IssuerSigningKey and TokenDecryptionKey.

Every single time I try to authorize the weather request it came back Unauthorized so I dug a little deeper by manually trying to handle the token validation. Now I am receiving this error which I think is why every request is coming back Unauthorized: Microsoft.IdentityModel.Tokens.SecurityTokenKeyWrapException: 'IDX10618: Key unwrap failed using decryption Keys

Our token service is generating the tokens by signing it with a Public Key in XML format and is then encrypting the token with a Private Key also in XML format. So this is how I am trying to validate the token in the new DotNet 6 Web API as following:

var builder = WebApplication.CreateBuilder(args);
var config = builder.Configuration;

// the signing key is a Public Key from an X509Certificate in XML format
RSACryptoServiceProvider signingRsa = new RSACryptoServiceProvider();
signingRsa.FromXmlString(config["JWT:SigningKey"]);
// I have also tried omitting the following two lines of code
byte[] publicKey = signingRsa.ExportRSAPublicKey();
signingRsa.ImportRSAPublicKey(publicKey, out _);

// the encryption key is a Private Key from an X509Certificate in XML format
RSACryptoServiceProvider encryptionRsa = new RSACryptoServiceProvider();
encryptionRsa.FromXmlString(config["JWT:EncryptionKey"]);
// I have also tried omitting the following two lines of code
byte[] privateKey = encryptionRsa.ExportRSAPrivateKey();
encryptionRsa.ImportRSAPrivateKey(privateKey, out _);

// try to read the token
string token = "[JWE Token]";
JwtSecurityTokenHandler handler = new JwtSecurityTokenHandler();
var jwt = handler.ReadJwtToken(token);

// validate the token
SecurityToken jwtToken;
// the error is thrown here!!!
var decryptedToken = handler.ValidateToken(token, new TokenValidationParameters
{
    ValidateIssuer = false,
    ValidateAudience = false,
    ValidateLifetime = false,   // this needs to be added back in once we integrate refresh tokens
    ValidateIssuerSigningKey = false,
    ValidIssuer = config["JWT:Issuer"],
    ValidAudience = config["JWT:Audience"],
    IssuerSigningKey = new RsaSecurityKey(signingRsa),
    TokenDecryptionKey = new RsaSecurityKey(encryptionRsa)
}, out jwtToken);

Any feedback or ideas is greatly appreciated!