Questions tagged [distributed-tracing]

Distributed Tracing aims to provide better observability into distributed systems and microservices for purposes of performance monitoring and troubleshooting issues.

Distributed Tracing

Distributed Tracing aims to provide better observability into distributed systems and microservices for purposes of performance monitoring and troubleshooting issues.

Modern Internet services are often implemented as complex, large-scale distributed systems. These applications are constructed from collections of software modules that may be developed by different teams, perhaps in different programming languages, and could span many thousands of machines across multiple physical facilities. Tools that aid in understanding system behavior and reasoning about performance issues are invaluable in such an environment.

Source: Dapper, a Large-Scale Distributed Systems Tracing Infrastructure

How it works in a nutshell

Distributed Tracing works by collecting the various entry and exit points and useful intermediate data and metrics done by a request until the final response is served to the requesting end. Some Distributed Tracing systems collect this information fully automatic while some other require manual instrumentation of code.

When entering a system, the request is usually assigned a unique Trace ID. This ID is then propagated to any participating systems. Information gathered this way is sent to some sort of backend collecting the data. The collector then aggregates the data via the Trace ID, thus showing the full request as it passed through the distributed system.

Metrics usually included are request time, latency, errors, status codes, etc. but not limited to this.

Open Source implementations:

Several Open Source implementations for Distributed Tracing exist:

  • http://opencensus.io

    A single distribution of libraries for metrics and distributed tracing with minimal overhead that allows you to export data to multiple backends.

  • http://opentracing.io

    Vendor-neutral APIs and instrumentation for distributed tracing

  • http://zipkin.io

    Zipkin is a distributed tracing system. It helps gather timing data needed to troubleshoot latency problems in microservice architectures. It manages both the collection and lookup of this data.

  • http://www.jaegertracing.io

    Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and troubleshooting microservices-based distributed systems

There is also a W3 working group aiming to standardize context propagation across various Distributed Tracing systems:

Because Distributed Tracing is crucial for application performance monitoring, most APM vendors adopted it in one way or another. Notable APM vendors offering Distributed Tracing are AppDynamics, DynaTrace, Instana, Lightstep or New Relic.

219 questions
1
vote
0 answers

Difference between the headers x-correlation-id, request-id, request-context and traceparent

I have an angular application calling server api's. I have enabled distrubuted tracing using application insights sdk "@microsoft/applicationinsights-web": "^2.5.4" enabling W3C tracing mode and see the End-To-End request trace in Azure Application…
1
vote
0 answers

Rename Apollo Federation's OpenTelemetry traces based on operation name (named queries)

I have a federated Apollo Graph (using ApolloGateway) and a bunch of subgraph microservices. I’ve set them up to do distributed tracing via OpenTelemetry. Now, they do produce a bunch of traces that I can see in my backend (Elastic APM). However,…
1
vote
1 answer

Create an open-telemetry span using trace-id and span-id in Java

I'm using open-telemetry to trace my applications and have a few microservices and Kafka broker in my distributed system. I'm using Java/spring-boot and in the HTTP Headers sending the traceparent(Ex:…
1
vote
1 answer

SpringBoot Sleuth/Zipkin Tracing: Issue adding a client, for Zipkin Server

Small question regarding a Java 11 + SpringBoot 2.6.7 web application, in the context of Zipkin please. I have a very simple micro services architecture SpringBoot repo that can be found here:…
PatPanda
  • 3,644
  • 9
  • 58
  • 154
1
vote
1 answer

sleuth trace id not getting printed for concurrent calls

In the log, for concurrent calls trace and span id are not getting printed. But for other calls, I can see it's getting printed. Below is my code. @RestController public class GetStudent implements IGetStudent { private static final Logger log…
1
vote
1 answer

OpenTelemetry is not tracing SQL Statements while using cursor_factory as NamedTupleCursor

Kindly look at the code below. I'm using opentelemetry for tracing. Psycopg2Instrumentor for PostgreSQL tracing. Here only the "show server_version" SQL statement is getting traced. But the SQL statement in execute method is not traced. I think it's…
1
vote
0 answers

How can I view a summary(average time of individual services) of all traces over zipkin containing 3 services(with node js)?

I have a node js application which includes 3 API calls (3 services). I am sending the traces to zipkin UI. There I can view individual traces and time spans of particular services. But I want to view the average time of any of the 3 services over a…
1
vote
1 answer

Opentelemetry context propagation test

I'm trying to test some opentelemetry spans are correctly build and linked in parent child relations. clientSpan = Span.wrap( SpanContext.createFromRemoteParent( "12345678123456781234567812345678", …
AdrianS
  • 1,980
  • 7
  • 33
  • 51
1
vote
0 answers

OpenCensus distributed tracing propagation using both Flask & requests

I'm attempting to implement distributed tracing via OpenCensus across different services where each service is using Flask for serving and sends downstream requests using... er.. requests to build its reply. The tracing platform is GCP Cloud…
1
vote
1 answer

Java Brave: .spanReporter() deprecated in Tracing.newBuilder().localServiceName("service").spanReporter(reporter).build();

Small question regarding Java Brave please. I have a very small piece of code: OkHttpSender sender = OkHttpSender.newBuilder().endpoint("https://zipkin-instance.com:9411/api/v2/spans").build(); AsyncReporter reporter =…
PatPanda
  • 3,644
  • 9
  • 58
  • 154
1
vote
2 answers

How configure properly Grafana Tempo?

I tried to use Grafana Tempo for distributed tracing. I launch it from docker-compose: version: "3.9" services: # MY MICROSERVICES ... prometheus: image: prom/prometheus ports: - ${PROMETHEUS_EXTERNAL_PORT}:9090 volumes: …
1
vote
0 answers

Propagate headers in Java microservice through Agent

We have many microservices in the system. I want to build a Java library to propagate auth tokens to each service request. I can intercept HHTP headers and write a custom interceptor for each client like RestTemplate, WebClient to inject headers.…
1
vote
0 answers

Issue in Koa.js for jaeger tracing with Istio

I am facing issue in jaeger tracing with Koa.js microservices.I haven't change anything regarding jaeger at my code level .I only install istio at AKS cluster and internally its taking tracing from there.But its showing tracing between two…
Sunil
  • 11
  • 2
1
vote
1 answer

What are my options for viewing the traces asp.net core captures in my applications?

I've recently been reading up on the topic of distributed tracing in asp.net core. I'm using asp.net core 5 and can see that it is automatically creating tracing information that is compatible with W3C context tracing. I plan to use this information…
omatase
  • 1,551
  • 1
  • 18
  • 42
1
vote
1 answer

Track python simpleHttp server logging information in azure application insights application map

We have different microservices(function apps, vm servers, etc) logging to application insights. A simple python http server is hosted on a linux VM, I want this server to receive a traceparent http header (W3C tracing) log the information to…