1

I have three services which subscribe and publish to separate queues on azure service bus. I've just added open telemetry for these services and it doesn't appear to propagate the trace ID between services.

For example, a single trace would ideally include the flow between services:

Service 1 -> Queue1 -> Service 2 -> Queue2 -> Service 3

It looks like I'm supposed to Inject/Extract this information between services, but there appears to be limited documentation/examples on how to do this, because apparently it's supposed to "just work" using the W3C Trace Context?

If anyone can provide any insight or pointers into exactly what I should be doing in order to achieve the desired outcome, that would be greatly appreciated.

Many thanks.

Darren
  • 1,071
  • 1
  • 15
  • 39

1 Answers1

0

I remember doing something similar with OpenCensus trace library when I needed to propagate traces between multiple GRPC services. I took the existing trace span, fed it to Binary function, got the bytes, set it as data on the outgoing context, then on the receiving side I extracted the trace span from the incoming context and fed it to trace.StartSpanWithRemoteParent.

This is the implementation of the util that was adding trace to context:

package grpcutil

import (
    "context"
    "encoding/base64"

    "github.com/grpc-ecosystem/go-grpc-middleware/util/metautils"
    "go.opencensus.io/trace"
    "go.opencensus.io/trace/propagation"
    "google.golang.org/grpc/metadata"
)

const TraceContextMDKey = "trace_ctx"

func NewTraceableContext(ctx context.Context, span *trace.Span) context.Context {
    return metadata.AppendToOutgoingContext(
        ctx,
        TraceContextMDKey, base64.StdEncoding.EncodeToString(propagation.Binary(span.SpanContext())),
    )
}

func GetRemoteTraceContext(ctx context.Context) (trace.SpanContext, bool) {
    var traceCtx []byte
    extCtx := metautils.ExtractIncoming(ctx).Get(TraceContextMDKey)
    if len(extCtx) > 0 {
        traceCtx, _ = base64.StdEncoding.DecodeString(extCtx)
    }
    return propagation.FromBinary(traceCtx)
}

How it looked on the receiving side:

remoteTraceCtx, _ := grpcutil.GetRemoteTraceContext(ctx)
_, span := trace.StartSpanWithRemoteParent(ctx, "name", remoteTraceCtx)
defer span.End()

Hope this points you in the right direction.

Artūrs Lataks
  • 188
  • 7
  • Hi, thanks. Yes I was wondering if I just need to add it myself. I don’t think the servicebus library I’m using currently supports it. I’m not entirely sure how to extract the propagation information from the context — I’ll look into it. Thanks for your input. – Darren Feb 24 '22 at 07:14
  • From all of the Go documentation regarding OpenTelemetry, it would seem that it _just works_ if you properly set up propagation, i.e. `otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))`, however, I've been struggling to get this working as well. It feels like this should be baked, but I can't get it working and docs and examples are very limited. – clownbaby Apr 06 '22 at 18:21