During a recent assignment, I was tasked to test out OpenTelemetry as a way to enhance visibility in our (ASP.NET Core) app. One of the goals I was hoping to achieve involved providing function-level traces throughout each of our services, but a concern we had was the amount of logging boilerplate code this would leave.
DispatchProxy has proven to be a useful tool here, but its downside so far is that the proxy will only be used for the first (outer) function call. Once within a function, all method calls from within that function go through the proxied object rather than the proxy, leaving no trace of these inner function calls. All time spent within the function is either attributed to the parent (when synchronous) or mysteriously missing from the tree (when async).
To demonstrate this issue, I have created a test project which you can clone and run for yourself to follow along with my example. I have left the Honeycomb exporter configuration helper commented out in ServiceCollectionExtensions.ConfigureTelemetry() for those who wish to use that visualization.
From Worker.cs:
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
// wait for the console to load before we report any activities
await Task.Delay(3000);
using var activity = Telemetry.ActivitySource.StartActivity("Worker Actions");
this.mainService.MethodWithNestedMethod();
this.mainService.CallSecondaryServiceNestedMethod();
this.lifetime.StopApplication();
}
In this image captured from Honeycomb, you can see the 'Worker Actions' activity output from the above code snippet. We see the first call to each proxied service, but any further calls made within those methods are skipping the proxy, and thus our handy logging hook is rendered useless. I've noted the areas where I would've expected a trace for the nested methods to appear.
What I am hoping to determine is if there is a way to instrument my services to capture these inner calls without resorting to manual instrumentation in every single method, whether it is through some existing proxy class like DispatchProxy, an OpenTelemetry helper I've yet to discover, or some other method entirely. We may decide against adopting a paid service if that's the only option, but it would be good to know anyway.
