Can jaeger be used to trace an end-to-end transaction with polling & callback used by the underlying components fulfilling steps of the transaction

Question

I'm a newbie on Jaeger and would like to know if I could trace an end-to-end transaction with a parent span & child ones like the one described below with polling from child components (no direct invocation from parent to child) and callback from childs to the parent component.

Context:

First let's describe a simplified view of what I'd like to do. A solution made of several components exposes a REST API to submit transactions. It synchronously returns a transaction Id after invocation and will callback the invoker upon completion or failure of the transaction. So what I want to trace is the overall transaction from 1 to 3:

invoker calls solution REST API with transaction input parameters & a callback url
invoker receives an Id from REST API response
solution call the invoker callback url with that Id

Under the hood:

the component (C1) exposing the REST API to submit transaction will perform some actions and then place the transaction in a queue.This is the parent component I was referring to above.
a second component (C2) will poll C1 using an internal REST API to fetch the transaction from the queue and then invoke the REST API exposed by a third component (C3) to further execute actions related to the transaction. In the message C1 propagates a callback url which is then propagated by C2 to C3.
the third component (C3) will inform C1 that it starts to process the transaction by invoking the propagated callback to C1 with IN-PROGRESS
and then C3 orchestrates many actions invoking synchronously (request<->response) or asynchronously (request<->response & then <-callback) different other components
When all actions are completed successfully or when one fails with a un-recoverable error, then component C3 will callback back again C1 to indicate SUCCESS or FAILURE.
When C1 is invoked by C3, it then performs some closing actions and invokes the callback of the transaction invoker.

I'm assuming that it will be possible to trace the entire transaction with Jaeger, but here are my questions: Question:

Q1: Am I right ? :-)
Q2: How to convey/propagate the parent span created in C1 to C2 ? (between step 1 & 2). I'm thinking at adding the SpanContext in a map as a new attribute of the transaction ?
Q3: Assuming C2 got the span it can propagate it to C3 and as C3 will invoke several times the callback to C1 (steps 3 & 5) how to correlate these callbacks invocation in C1 with the parent span ?

Any hints & tips will be welcome. Thx.

fckbo · Answer 1 · 2020-06-01T22:34:32.213

so I give it a try and my answer to the question above are:

Q1) yes it is possible (congrats to the jaeger team, package pretty easy to grasp with a good documentation)
Q2) I did struggle a bit with this one and thanks to https://github.com/CHOMNANP/jaeger-js-text-map-demo I implemented a solution by adding a "textCarrier" with a ref. to the span context formatted as "FORMAT_TEXT_MAP" to the message Component 1 was publishing towards Component 2.

Code snipper in C1 on the first API invocation

server.post("/api/vms", (req, res) => {
  console.log('Enter /api/vms');
  const span = tracer.startSpan(req.path);
  // Use the log api to capture a log
  span.log({ event: 'request_received' })
  txSpan = span;
  //console.log("req",span);
  // Use the setTag api to capture standard span tags for http traces
  span.setTag(opentracing.Tags.HTTP_METHOD, req.method)
  span.setTag(opentracing.Tags.SPAN_KIND, opentracing.Tags.SPAN_KIND_RPC_SERVER)
  span.setTag(opentracing.Tags.HTTP_URL, req.path)

followed by this part when sending the msg on redis:

   const textCarrier = getTextCarrierBySpanObject(span);
   tracer.inject(span.context(), opentracing.FORMAT_TEXT_MAP, textCarrier)

   var vm = req.body;
   console.log('Creating a new vm: ', vm);
   // Publish a message by specifying a channel name.
   try {
      Object.assign(vm, { textCarrier });
      pub.publish('tasks-queue', JSON.stringify(vm));
   } catch(e) {
        console.log("App1 Error when publishing task to App2", e);
   }

The getTextCarrierBySpanObject function is coming from https://github.com/CHOMNANP/jaeger-js-text-map-demo

function getTextCarrierBySpanObject(_span) {

    const spanContext = _span.context();
    const traceId = spanContext._traceId.toString('hex');
    const spanId = spanContext._spanId.toString('hex');
    let parentSpanId = spanContext._parentId;
    const flag = _.get(spanContext, '_flags', 1);

    if (parentSpanId) {
        parentSpanId = parentSpanId.toString('hex');
    } else {
        parentSpanId = 0;
    }

    const uberTraceId = `${traceId}:${spanId}:${parentSpanId}:${flag}`;
    console.log("uberTraceId===> ", uberTraceId)

    let textCarrier = {
        "uber-trace-id": uberTraceId
    };

    return textCarrier
}

Code snippet in C2 receiving the msg from redis

sub.on('message', function(channel, message) {
  // message is json string in our case so we are going to parse it.
  try {
    var json = JSON.parse(message)
    console.log("Task received", message);

    const tracer = opentracing.globalTracer();
    // Extracting the span context from the message
    var parentSpan = tracer.extract(opentracing.FORMAT_TEXT_MAP, JSON.parse(message).textCarrier);
    console.log("textCarrier=",JSON.parse(message).textCarrier);
    const span = tracer.startSpan("/msg", { childOf: parentSpan });
    // Use the log api to capture a log
    span.log({ event: 'msg_received' })

I tested with version 1.13

docker run -d --name jaeger   -e COLLECTOR_ZIPKIN_HTTP_PORT=9411   -p 5775:5775/udp   -p 6831:6831/udp   -p 6832:6832/udp   -p 5778:5778   -p 16686:16686   -p 14268:14268   -p 9411:9411   jaegertracing/all-in-one:1.13

Q3) This one is pretty straight forward using FORMAT_HTTP_HEADERS to convey the span from the component C4 invoking a callback on C3 and the from the Component C3 invoking a callback on C1. The only "problem" I found was more a "trace readability issue" as, in fact, the Spans appear on the Jaeger UI in "the progagation" order and not in the "timing order" which can be a bit confusing... but the "trace graph" experimental feature allowed to actually see the trace in the right order of appearance, so all good.

So all in all a pretty convincing prototyping exercise with Jaeger, will most probably pilot it now on a real project before trying it in production.

Can jaeger be used to trace an end-to-end transaction with polling & callback used by the underlying components fulfilling steps of the transaction

Context:

1 Answers1