How to add tracing to a Rust microservice?

Question

I built a microservice in Rust. I receive messages, request a document based on the message, and call a REST api with the results. I built the REST api with warp and send out the result with reqwest. We use jaeger for tracing and the "b3" format. I have no experience with tracing and am a Rust beginner.

Question: What do I need to add the the warp / reqwest source below to propagate the tracing information and add my own span?

My version endpoint (for simplicity) looks like:

pub async fn version() -> Result<impl warp::Reply, Infallible> {
    Ok(warp::reply::with_status(VERSION, http::StatusCode::OK))
}

I assume I have to extract e.g. the traceid / trace information here.

A reqwest call I do looks like this:

pub async fn get_document_content_as_text(
    account_id: &str,
    hash: &str,
) -> Result<String, Box<dyn std::error::Error>> {

    let client = reqwest::Client::builder().build()?;
    let res = client
        .get(url)
        .bearer_auth(TOKEN)
        .send()
        .await?;
    if res.status().is_success() {}
    let text = res.text().await?;
    Ok(text)
}

I assume I have to add the traceid / trace information here.

Answer 1

I'll assume that you're using tracing within your application and using opentelemetry and opentelemetry-jaeger to wire it up to an external service. The specific provider you're using doesn't matter. Here's a super simple setup to get that all working that I'll assume you're using on both applications:

# Cargo.toml
[dependencies]
opentelemetry = "0.17.0"
opentelemetry-jaeger = "0.16.0"
tracing = "0.1.33"
tracing-subscriber = { version = "0.3.11", features = ["env-filter"] }
tracing-opentelemetry = "0.17.2"

reqwest = "0.11.11"
tokio = { version = "1.21.1", features = ["macros", "rt", "rt-multi-thread"] }
warp = "0.3.2"

opentelemetry::global::set_text_map_propagator(opentelemetry_jaeger::Propagator::new());
tracing_subscriber::registry()
    .with(tracing_opentelemetry::layer().with_tracer(
        opentelemetry_jaeger::new_pipeline()
            .with_service_name("client") // or "server"
            .install_simple()
            .unwrap())
   ).init();

---

Let's say the "client" application is set up like so:

#[tracing::instrument]
async fn call_hello() {
    let client = reqwest::Client::default();
    let _resp = client
        .get("http://127.0.0.1:3030/hello")
        .send()
        .await
        .unwrap()
        .text()
        .await
        .unwrap();
}

#[tokio::main]
async fn main() {
    // ... initialization above ...

    call_hello().await;
}

The traces produced by the client are a bit chatty because of other crates but fairly simple, and does not include the server-side:

---

Let's say the "server" application is set up like so:

#[tracing::instrument]
fn hello_handler() -> &'static str {
    tracing::info!("got hello message");
    "hello world"
}

#[tokio::main]
async fn main() {
    // ... initialization above ...

    let routes = warp::path("hello")
        .map(hello_handler);

    warp::serve(routes).run(([127, 0, 0, 1], 3030)).await;
}

Likewise, the traces produced by the server are pretty bare-bones:

---

The key part to marrying these two traces is to declare the client-side trace as the parent of the server-side trace. This can be done over HTTP requests with the traceparent and tracestate headers as designed by the W3C Trace Context Standard. There is a TraceContextPropagator available from the opentelemetry crate that can be used to "extract" and "inject" these values (though as you'll see, its not very easy to work with since it only works on HashMap<String, String>s).

For the "client" to send these headers, you'll need to:

• get the current tracing Span
• get the opentelemetry Context from the Span (if you're not using tracing at all, you can skip the first step and use Context::current() directly)
• create the propagator and fields to propagate into and "inject" then from the Context
• use those fields as headers for reqwest

#[tracing::instrument]
async fn call_hello() {
    let span = tracing::Span::current();
    let context = span.context();
    let propagator = TraceContextPropagator::new();
    let mut fields = HashMap::new();
    propagator.inject_context(&context, &mut fields);
    let headers = fields
        .into_iter()
        .map(|(k, v)| {(
            HeaderName::try_from(k).unwrap(),
            HeaderValue::try_from(v).unwrap(),
        )})
        .collect();

    let client = reqwest::Client::default();
    let _resp = client
        .get("http://127.0.0.1:3030/hello")
        .headers(headers)
        .send()
        .await
        .unwrap()
        .text()
        .await
        .unwrap();
}

For the "server" to make use of those headers, you'll need to:

• pull them out from the request and store them in a HashMap
• use the propagator to "extract" the values into a Context
• set that Context as the parent of the current tracing Span (if you didn't use tracing, you could .attach() it instead)

#[tracing::instrument]
fn hello_handler(traceparent: Option<String>, tracestate: Option<String>) -> &'static str {
    let fields: HashMap<_, _> = [
        dbg!(traceparent).map(|value| ("traceparent".to_owned(), value)),
        dbg!(tracestate).map(|value| ("tracestate".to_owned(), value)),
    ]
    .into_iter()
    .flatten()
    .collect();

    let propagator = TraceContextPropagator::new();
    let context = propagator.extract(&fields);
    let span = tracing::Span::current();
    span.set_parent(context);

    tracing::info!("got hello message");
    "hello world"
}

#[tokio::main]
async fn main() {
    // ... initialization above ...

    let routes = warp::path("hello")
        .and(warp::header::optional("traceparent"))
        .and(warp::header::optional("tracestate"))
        .map(hello_handler);

    warp::serve(routes).run(([127, 0, 0, 1], 3030)).await;
}

With all that, hopefully your traces have now been associated with one another!

Full code is available here and here.

---

^{Please, someone let me know if there is a better way! It seems ridiculous to me that there isn't better integration available. Sure some of this could maybe be a bit simpler and/or wrapped up in some nice middleware for your favorite client and server of choice... But I haven't found a crate or snippet of that anywhere!}

Answer 2

You need to add a tracing filter into your warp filter pipeline.

From the documentation example:

use warp::Filter;

let route = warp::any()
    .map(warp::reply)
    .with(warp::trace(|info| {
        // Create a span using tracing macros
        tracing::info_span!(
            "request",
            method = %info.method(),
            path = %info.path(),
        )
    }));