AWS XRay outgoing tracing for ASP.NET WebAPI

Question

Environment

• ASP.NET WebAPI calling external client services.

• XRay incoming trace applied in Global.asax via

  AWSXRayASPNET.RegisterXRay(this, "CustomerAPI");
  

• HttpClient instance hidden as private member in utility class, shared in a separate project with multiple WebAPI

• HttpClient having delegating handler for Xray enable as follows:- (Ref: .NET HttpClient for outgoing XRay)

  bool isXrayEnabled = false;                    
  bool.TryParse(Config["XRayEnable"], out isXrayEnabled);
  
  if (isXrayEnabled)
  {
       Logger.Info("Enabling XRay tracing with Http calls");
       httpClient = new HttpClient(new HttpClientXRayTracingHandler(new HttpClientHandler()));
  }
  else
  {
       httpClient = new HttpClient();
       Logger.Info("Disabling XRay tracing with Http calls");
  }
  

Issue While all APIs shows incoming XRay trace, the outgoing trace is not shown at all.

Instead, following error is shown in the XRay logs

   Exception type: EntityNotAvailableException

    Exception message: Entity doesn't exist in HTTPContext
   at Amazon.XRay.Recorder.Core.Internal.Context.HybridContextContainer.InjectEntityInTraceContext()
   at Amazon.XRay.Recorder.Core.AWSXRayRecorderImpl.AddHttpInformation(String key, Object value)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Amazon.XRay.Recorder.Core.AWSXRayRecorderImpl.AddHttpInformation(String key, Object value)
   at Amazon.XRay.Recorder.Handlers.AspNet.AWSXRayASPNET.ProcessHTTPResponse(Object sender, EventArgs e)
   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Answer 1

From the error stack it looks like the exception occurs here :https://github.com/aws/aws-xray-sdk-dotnet/blob/master/sdk/src/Handlers/AspNet/AWSXRayASPNET.cs#L209. The response received in this event doesn't have Segment started by the middleware in the BeginRequest : https://github.com/aws/aws-xray-sdk-dotnet/blob/master/sdk/src/Handlers/AspNet/AWSXRayASPNET.cs#L141 and hence not able to store HTTP information.

Can you answer following questions:

1. How is the flow for a given incoming request : request->WEBAPI->HTTPClient? Are the HTTPClient calls asynchronous?
2. If you have ASP.NET middleware instrumented and do not enable HTTPClient X-Ray instrumentation, do you still get above error? Or this error is a result of enabling HTTPClient instrumentation. Reason to narrow down since the error stack is coming from X-Ray middleware and not HTTP client.
3. HttpClientXRayTracingHandler creates a Subsegment for you. This instrumentor expects Segment to be already present in the TraceContext which in this case is created by AWSXRayASPNET. As per your statement, "HttpClient instance hidden as private member in utility class, shared in a separate project with multiple WebAPI", can you check whether Segment is present in the TraceContext just before making a network call using AWSXRayRecorder.Instance.GetEntity(). This should return Segment started by the middleware or subsegment if you have manually created before this call.
4. Do you have any configuration related to threadpool leading to TraceContext not correctly propagated between threads?

In the above code snippet, you don't need to to do "isXrayEnabled()" check. By default you can instrument HTTPClient with X-Ray ant the middleware does this check for you : https://github.com/aws/aws-xray-sdk-dotnet/blob/master/sdk/src/Handlers/System.Net/Utils/RequestUtil.cs#L59

Thanks, Yogi