1

I am interested in using Elastic APM within an ASP.NET Core to instrument traces of a set of services which communicate over a mix of protocols (HTTP, SQS, SNS). Despite reviewing the documentation, I am not clear how I can use the Elastic APM Public API to connect transactions to one another which occur outside of HTTP (HttpClient is automatically instrumented for trace by Elastic APM).

According to the documentation, I should be able to serialize the CurrentTransaction.OutgoingDistributedTracingData on the caller and then deserialize it to resume the transaction on the callee, but despite implementing this pattern in memory, my traces in Kibana are missing spans from all but the final transaction. 

// transaction 1
var trans1 = Agent.Tracer.StartTransaction("Dist Trans 2", ApiConstants.TypeRequest);

await trans1.CaptureSpan("step 1 processing", ApiConstants.ActionExec, async () => await Task.Delay(30));

// transaction 2
var trans2 = Agent.Tracer.StartTransaction("Dist Trans 2", ApiConstants.TypeRequest,
    DistributedTracingData.TryDeserializeFromString(trans1.OutgoingDistributedTracingData.SerializeToString()));

await trans2.CaptureSpan("step 2 processing", ApiConstants.ActionExec, async () => await Task.Delay(30));

// transaction 3
var trans3 = Agent.Tracer.StartTransaction("Dist Trans 2", ApiConstants.TypeRequest,
    DistributedTracingData.TryDeserializeFromString(trans2.OutgoingDistributedTracingData.SerializeToString()));

await trans3.CaptureSpan("step 3 processing", ApiConstants.ActionExec, async () => await Task.Delay(30));

trans3.End();

kibana trace dashboard

My implementation spike can be found on Github.

Nathan Taylor
  • 24,423
  • 19
  • 99
  • 156

1 Answers1

3

You don't end trans1 and trans2.

Just put these 2 lines to the point where these end, and everything should show up fine: 

trans1.End();
trans2.End();

There is the CaptureTransaction, which is a convenient method that can wrap you any code and makes sure the transaction is ended and all exceptions are captured - so you use that method and it does "everything" for you.

Then there is the StartTransaction method - this is the one you use in your code -, which starts the transaction and does not do anything else. The advantage here is that you get an ITransaction instance which you can use wherever and whenever you want. But in this case you need to call .End() on it manually once the transaction (aka the code you want to capture) is executed.

Same with CaptureSpan and StartSpan.

So you used CaptureSpan for your spans, so those where ended automatically when the lambda with Task.Delay finished, on the other hand you started your transactions with StartTransaction but only called .End() on trans3 and not on the 2 other transactions.

There is some explanation with a demo here - sample code of that demo is here.

gregkalapos
  • 3,529
  • 2
  • 19
  • 35
  • I experimented with this originally. It was counter intuitive to me to "end" the transaction which I was trying to extend or continue. When I did it the first time, I must have screwed it up because traces were not being connected. That said, trying it again now, and my traces are working for both my examples in the linked repo. Awesome. – Nathan Taylor Mar 27 '20 at 19:24
  • 1
    That's a good point: so it's better to think about this in a way that you don't really extend or continue the transaction. The terminology Elastic APM uses are: Trace, Transaction, Span. The 3 transactions here are normal transactions that you end normally, but they together form a trace - so you don't "continue" the transactions, you just make sure they are part of the same trace. Some hopefully helpful docs here: https://www.elastic.co/guide/en/apm/get-started/current/distributed-tracing.html – gregkalapos Mar 27 '20 at 19:33