Opentelemetry + X-Ray + Flask + ECS - not instrumenting endpoints

Question

I have a problem with instrumenting my Flask application with OpenTelemetry and XRay.

1. I've configured my main.py

   propagate.set_global_textmap(AwsXRayPropagator())
   span_processor_to_aws = BatchSpanProcessor(OTLPSpanExporter())
   trace.set_tracer_provider(
       TracerProvider(
           id_generator=AwsXRayIdGenerator(),
           resource=get_aggregated_resources(
               [
                   AwsEcsResourceDetector(),
               ]
           ),
       )
   )
   trace.get_tracer_provider().add_span_processor(span_processor_to_aws)
   
   FlaskInstrumentor().instrument_app(app)
   
   

2. In main.py I've added for tests:

tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("My Custom Trace in main") as curr_span_with_events:
    curr_span_with_events.set_attribute("Test_msg_1", "Test span msg 1 in main")

3. In my endpoint I've added:

   with tracer.start_as_current_span("internal_span_in_endpoint") as endpoint_span:
       endpoint_span.set_attribute("endpoint_event_1", ...)

4. I am using ADOT as a sidecar service for my API service, so I extended task-definition

      "name": "otel-collector",
      "image": "amazon/aws-otel-collector:latest",
      "cpu": 32,
      "memoryReservation": 256,
      "portMappings": [
        {
           "containerPort": 2000,
           "protocol": "udp"
        },
        {
            "containerPort": 4317,
            "protocol": "grpc"
        },
        {
            "containerPort": 55681,
            "protocol": "http"
        }
      ],

5. And also extended API task-definition with env variables:

        "name": "OTEL_EXPORTER_OTLP_ENDPOINT",
        "value": "http://localhost:55681"
      },
      {
        "name": "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT",
        "value": "http://localhost:55681/v1/traces"
      },
      {
        "name": "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT",
        "value": "http://localhost:55681/v1/metrics"
      },

With all this I have:

1. When API is build - I see 1 trace appears in X-Ray dashboard - it is from main.py (My Custom Trace in main)
2. When I make a call to an endpoint where I've manually declared a span - I don't see this trace shown in X-Ray Dashboard
3. When I make a call to any other endpoint - I don't see traces also (I assume all of endpoints should be instrumented by FlaskInstrumentor().instrument_app(app) but they are not).

What am I doing wrong here?

Answer 1

I think you might be missing setting the global propagator. Propagator propagates the trace header and without setting it you won't be able to see other traces. Here's some instruction for you to set propagator: https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/propagator/opentelemetry-propagator-aws-xray