How to use Rusts tracing_distributed

Question

I am trying to use the Rust tracing_distributed package, but I am getting strange and unhelpful errors when using it, and I am assuming I am using it wrong, but there is no documentation and there are no examples about how to use it. Here is an example of what I'm trying to do :

let trace = tracing_distributed::register_dist_tracing_root(traceId, remote_parent_span_id));
println!("trace value: {:?}", trace);

// the result of trace is: Err(NoEnabledSpan)

I have tried passing a few things in as the traceID and remote_parent_span_id including:

traceId = remote_parent_span_id = Some(tracing::Span::current())

As well as:

traceId = Some(tracing::Span::current())
remote_parent_span_id = ~someParentRequestIdGeneratedUpstream~

I know that the current span is not disabled from trying:

let span = tracing::Span::current();
if span.is_disabled() {
    println!("CURRENT SPAN DISABELED");
}

So this leads me to think that the issue is coming from not having the subscriber set properly. I am trying to set the subscriber in an init function which is called before this function which looks like this:

let subscriber = tracing_subscriber::registry() // provide underlying span data store
        .with(
            tracing_subscriber::fmt::layer()
                .json()
                .with_span_events(FmtSpan::ACTIVE)
                .event_format(stackdriver::StackDriverEventFormat::default())
                .with_filter(tracing_subscriber::filter::dynamic_filter_fn(
                    move |m, c| filter_layer.enabled(m, c.to_owned()),
                )),
        );

    let _ = tracing::subscriber::set_global_default(subscriber)
        .map_err(|_err| eprintln!("Unable to set global default subscriber"));

Would anyone be willing to provide me with an example of how to use this library? Or can anyone see what I'm doing wrong here? I have tried everything I can think of.

Answer 1

tracing-distributed has a test which demonstrates how to create and use TelemetryLayer.

I made a demo based on it. In this demo, NoEnabledSpan may be caused by missing #[instrument], which creates a Span for function foo. Hope this will help you find the actual cause.

Also tracing-honeycomb is a great use case for tracing-distributed, better check it out.

use std::sync::{Arc, Mutex};

use tracing::{Id, info};
use tracing::instrument;
use tracing_distributed::{Event, Span, Telemetry, TelemetryLayer};
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::registry;

#[derive(Default, Debug)]
pub struct BlackholeVisitor;

#[derive(PartialEq, Eq, Hash, Copy, Clone, Debug)]
pub struct TraceId(pub(crate) u128);

type SpanId = tracing::Id;

impl tracing::field::Visit for BlackholeVisitor {
    fn record_debug(&mut self, _: &tracing::field::Field, _: &dyn std::fmt::Debug) {}
}

/// Mock telemetry capability
pub struct TestTelemetry {
    spans: Arc<Mutex<Vec<Span<BlackholeVisitor, SpanId, TraceId>>>>,
    events: Arc<Mutex<Vec<Event<BlackholeVisitor, SpanId, TraceId>>>>,
}

impl TestTelemetry {
    pub fn new(
        spans: Arc<Mutex<Vec<Span<BlackholeVisitor, SpanId, TraceId>>>>,
        events: Arc<Mutex<Vec<Event<BlackholeVisitor, SpanId, TraceId>>>>,
    ) -> Self {
        TestTelemetry { spans, events }
    }
}

impl Telemetry for TestTelemetry {
    type Visitor = BlackholeVisitor;
    type TraceId = TraceId;
    type SpanId = SpanId;

    fn mk_visitor(&self) -> Self::Visitor {
        BlackholeVisitor
    }

    fn report_span(&self, span: Span<BlackholeVisitor, SpanId, TraceId>) {
        // succeed or die. failure is unrecoverable (mutex poisoned)
        let mut spans = self.spans.lock().unwrap();
        spans.push(span);
    }

    fn report_event(&self, event: Event<BlackholeVisitor, SpanId, TraceId>) {
        // succeed or die. failure is unrecoverable (mutex poisoned)
        let mut events = self.events.lock().unwrap();
        events.push(event);
    }
}

#[instrument]
fn foo() {
    let trace = tracing_distributed::register_dist_tracing_root(TraceId(123), Option::<Id>::None);
    println!("trace value: {:?}", trace);
    info!("test");
}

fn main() {
    let spans = Arc::new(Mutex::new(Vec::new()));
    let events = Arc::new(Mutex::new(Vec::new()));

    let cap = TestTelemetry::new(spans.clone(), events.clone());

    let telemetry_layer = TelemetryLayer::new("test_svc_name", cap, |x| x);
    let subscriber = registry::Registry::default()
        .with(tracing_subscriber::fmt::Layer::default())
        .with(telemetry_layer);
    // dbg!(&subscriber);
    tracing::subscriber::set_global_default(subscriber).expect("setting global default failed");
    foo();
    dbg!(&spans);
    dbg!(&events);
}

crate versions:

tracing = "0.1.32"
tracing-distributed = "0.4.0"
tracing-subscriber = "0.3.10"