Update from .NET Core 2.1 to .NET Core 3.1: Using Terraform 0.12.31 to build out the AWS Resources: AWS IoT Publish now throws SSL Error

Question

I am doing an upgrade from .NET Core 2.1 to .NET Core 3.1. I have upgraded my projects to use .NET Core 3.1. I am using the AWS IoT package described below

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <TargetFramework>netcoreapp3.1</TargetFramework>
        <LangVersion>latest</LangVersion>
    </PropertyGroup>

    <ItemGroup>
      <PackageReference Include="Amazon.Lambda.APIGatewayEvents" Version="2.4.0" />
      <PackageReference Include="Amazon.Lambda.Core" Version="2.1.0" />
      <PackageReference Include="Amazon.Lambda.Serialization.Json" Version="2.0.0" />
      <PackageReference Include="AWSSDK.ApiGatewayManagementApi" Version="3.7.0.96" />
      <PackageReference Include="AWSSDK.CognitoIdentityProvider" Version="3.7.1.68" />
      <PackageReference Include="AWSSDK.DynamoDBv2" Version="3.7.0.97" />
      <PackageReference Include="AWSSDK.IoT" Version="3.7.6.28" />
      <PackageReference Include="AWSSDK.S3" Version="3.7.4.3" />
      <PackageReference Include="Microsoft.IdentityModel.JsonWebTokens" Version="6.15.0" />
      <PackageReference Include="Microsoft.IdentityModel.Tokens" Version="6.15.0" />
      <PackageReference Include="Newtonsoft.Json" Version="13.0.1" />
      <PackageReference Include="System.IdentityModel.Tokens.Jwt" Version="6.15.0" />
    </ItemGroup>

</Project>

Now, when I use the AWS IoT package to publish a message I log this SSL Error

ERROR: Exception: Type: System.Net.Http.HttpRequestException Message: System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure. at System.Net.Security.SslStream.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception) at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslStream.PartialFrameCallback(AsyncProtocolRequest asyncRequest) --- End of stack trace from previous location where exception was thrown --- at System.Net.Security.SslStream.ThrowIfExceptional() at System.Net.Security.SslStream.InternalEndProcessAuthentication(LazyAsyncResult lazyResult) at System.Net.Security.SslStream.EndProcessAuthentication(IAsyncResult result) at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult) at System.Net.Security.SslStream.<>c.b__65_1(IAsyncResult iar) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location where exception was thrown --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken) at Amazon.Runtime.Internal.HttpHandler1.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext) at Jobs.Function.PostJobDiscussion(APIGatewayProxyRequest apiGatewayProxyRequest, ILambdaContext context) in /codebuild/output/src782906800/src/api/Jobs/src/Jobs/PostJobDiscussion.cs:line 98

I am using a Cloudfront distribution and I redirect my app to HTTPS for this Post api call.

I am building out these AWS resources using Terraform 0.12.31 My cloud front distribution uses an s3 bucket configuration for Terraform is described below

resource "aws_cloudfront_distribution" "s3_distribution" {
  origin {
    domain_name = aws_s3_bucket.webapp_s3.bucket_regional_domain_name
    origin_id   = "S3-${var.account_alias}-${var.domain_name}"
    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.portal_cdn_oai.cloudfront_access_identity_path
    }
  }

  enabled             = true
  is_ipv6_enabled     = true
  comment             = ""
  default_root_object = "index.html"

  // setup a custom error to default to the index.html for the spa
  custom_error_response {
    error_caching_min_ttl = 300
    error_code            = "404"
    response_code         = "200"
    response_page_path    = "/index.html"
  }

  aliases = [var.cdn_domain_name]

  .....

  viewer_certificate {
    cloudfront_default_certificate = false
    acm_certificate_arn            = data.aws_acm_certificate.portal_cert.arn
    minimum_protocol_version       = "TLSv1.1_2016"
    ssl_support_method             = "sni-only"
  }
}

I have checked that my certificates in AWS are valid meaning they are not expired but the error says

The remote certificate is invalid according to the validation procedure.

Somehow the SSL/TLS handshake is not working

After my upgrade to .NET 3.1 any lambda that is using AWS IoT Publish is failing.

In the browser, the response headers The Response Headers content-length: 41 content-type: application/json date: Wed, 08 Dec 2021 16:55:04 GMT via: 1.1 2d64784a9c03401d2f7f505cbef3c986.cloudfront.net (CloudFront) x-amz-apigw-id: KChz0HFCIAMFeWw= x-amz-cf-id: YSrTkIkUyMZY3wiT91WcmFDivk_0cmwlEUE0OeQtD3wGUREz4pXf7Q== x-amz-cf-pop: IAH50-C2 x-amzn-errortype: InternalServerErrorException x-amzn-requestid: a00f8e86-de54-44d4-8595-b87a50c04a4b x-cache: Error from cloudfront

The Request Headers :authority: api.dev2.project-caisson.com :method: POST :path: /jobs/c9b8d1d2-b68c-45d1-9fde-27fa8cc358dd/discussion :scheme: https accept: application/json, text/plain, / accept-encoding: gzip, deflate, br accept-language: en-US,en;q=0.9 authorization: jwt token content-length: 129 content-type: application/json origin: http://localhost:4200 referer: http://localhost:4200/ sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96" sec-ch-ua-mobile: ?0 sec-ch-ua-platform: "Windows" sec-fetch-dest: empty sec-fetch-mode: cors sec-fetch-site: cross-site user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36

The Signnature algorithm for the certificate is SHA256WITHRSA

The policy on my lambda for IoT is { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iot:Connect", "iot:Publish", "iot:Subscribe", "iot:Receive", "iot:GetThingShadow", "iot:UpdateThingShadow", "iot:DeleteThingShadow" ], "Resource": "*" } ] }

These certs have not changed since the upgrade. I have viewed the certs using openssl command and everything looks fine.

The code in C# that triggers the error is

 using (var iotDataClient = CreateAmazonIotDataClient())
                    {
                        var mqttBrokerEndpoint = Environment.GetEnvironmentVariable("MQTTBroker");
                        context.Logger.LogLine($"Created an AmazonIotDataClient: MQTT Env Variable {mqttBrokerEndpoint}");
                        var serviceURL = iotDataClient.Config.ServiceURL;
                        context.Logger.LogLine($"AmazonIotDataClient: ServiceURL {serviceURL}");

                        context.Logger.LogLine($"iotDataClient Config: ProxyHost {iotDataClient.Config.ProxyHost}");
                        context.Logger.LogLine($"iotDataClient Config: ProxyPort {iotDataClient.Config.ProxyPort}");
                        context.Logger.LogLine($"iotDataClient Config: AllowAutoRedirect {iotDataClient.Config.AllowAutoRedirect}");
                        context.Logger.LogLine($"iotDataClient Config: AuthenticationServiceName {iotDataClient.Config.AuthenticationServiceName}");
                        context.Logger.LogLine($"iotDataClient Config: Url {iotDataClient.Config.DetermineServiceURL()}");


                        //iotDataClient.Config.Validate();

                        // There are three levels of QoS: 0 - at most once. 1 - at least once. 2 - exactly once.
                        await iotDataClient.PublishAsync(new PublishRequest
                        {
                            Topic = $"{latestMessageFromMTT.ClientID}/JobDiscussion",
                            Qos = 1,
                            Payload = new MemoryStream(
                                Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(jobDiscussion)))
                        }, cancellationTokenSource.Token);
                    }

With the CreateAmazonIotDataClient()

if (_iotDataClient != null) return _iotDataClient;

            var mqttBrokerEndpoint = Environment.GetEnvironmentVariable("MQTTBroker");
            var clientConfig = new AmazonIotDataConfig { ServiceURL = mqttBrokerEndpoint };
            return new AmazonIotDataClient(clientConfig);

And the ServiceURL is https://.iot.us-east-1.amazonaws.com

How do I test that this broker is good. Are there any other tools I should be using to try and identify why I am getting SSL errors?