OpenTelemetry Export Traces to Elastic APM and Elastic OpenDistro

Question

I am trying to instrument by python app (django based) to be able to push transaction traces to Elastic APM which I can later view using the Trace Analytic in OpenDistro Elastic.

I have tried the following

Method 1:

pip install opentelemetry-exporter-otlp

Then, in the manage.py file, I added the following code to directly send traces to elastic APM.

    span_exporter = OTLPSpanExporter(
        endpoint="http://localhost:8200",
        insecure=True
    )

When I run the code I get the following error:

Transient error StatusCode.UNAVAILABLE encountered while exporting span batch, retrying in 1s.
Transient error StatusCode.UNAVAILABLE encountered while exporting span batch, retrying in 2s.

Method 2:

I tried using OpenTelemetry Collector in between since method 1 didn't work. I configured my collector in the following way:

extensions:
  memory_ballast:
    size_mib: 512
  zpages:
    endpoint: 0.0.0.0:55679

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
  memory_limiter:
    # 75% of maximum memory up to 4G
    limit_mib: 1536
    # 25% of limit up to 2G
    spike_limit_mib: 512
    check_interval: 5s

exporters:
  logging:
    logLevel: debug
  otlp/elastic:
    endpoint: "198.19.11.22:8200"
    insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [logging, otlp/elastic]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [logging]

  extensions: [memory_ballast, zpages]

And configured my code to send traces to collector like this -

    span_exporter = OTLPSpanExporter(
        endpoint="http://localhost:4317",
        insecure=True
    )

Once I start the program, I get the following error in the collector logs -

go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
    go.opentelemetry.io/collector@v0.35.0/exporter/exporterhelper/queued_retry.go:304
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
    go.opentelemetry.io/collector@v0.35.0/exporter/exporterhelper/traces.go:116
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
    go.opentelemetry.io/collector@v0.35.0/exporter/exporterhelper/queued_retry.go:155
go.opentelemetry.io/collector/exporter/exporterhelper/internal.ConsumerFunc.Consume
    go.opentelemetry.io/collector@v0.35.0/exporter/exporterhelper/internal/bounded_queue.go:103
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*BoundedQueue).StartConsumersWithFactory.func1
    go.opentelemetry.io/collector@v0.35.0/exporter/exporterhelper/internal/bounded_queue.go:82
2022-01-05T17:36:55.349Z    error   exporterhelper/queued_retry.go:304  Exporting failed. No more retries left. Dropping data.  {"kind": "exporter", "name": "otlp/elastic", "error": "max elapsed time expired failed to push trace data via OTLP exporter: rpc error: code = Unavailable desc = connection closed", "dropped_items": 1}

What am I possibly missing here?

NOTE: I am using the latest version of opentelemetry sdk and apis and latest version of collector.

opendistro is not Elasticsearch and it's not an Elastic product fwiw — warkolm, Jan 05 '22 at 21:11
Have you tried with Elasticsearch, rather than OpenDistro? I wonder if there's an incompatibility there. — Colton Myers, Jan 06 '22 at 15:56
haven't tried with elasticsearch. Would not be possible to do it. We are restricted by the ops team. — Anirudh Bagri, Jan 06 '22 at 17:55
i'm getting the same error you have in method 1 but i'm using grpc from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter i have been digging through this issue for two days straight, i don't understand what the problem is. My code was working perfectly fine before and suddenly it's not. — hashguard, Jan 26 '22 at 16:54

score 1 · Answer 1 · answered Jan 07 '22 at 16:41

Okay, So the way to work with Open Distro version of Elastic to get traces is:

To avoid using the APM itself. OpenDistro provides a tool called Data Prepper which must be used in order to send data(traces) from Otel-Collector to OpenDistro Elastic.

Here is the configuration I did for the Otel-Collector to send data to Data Prepper:

... # other configurations like receivers, etc. 
exporters:
  logging:
    logLevel: debug
  otlp/data-prepper:
    endpoint: "http://<DATA_PREPPER_HOST>:21890"
    tls:
      insecure: true
... # Other configurations like pipelines, etc.

And this is how I configured Data Prepper to receive data from Collector and send it to Elastic

entry-pipeline:
  delay: "100"
  source:
    otel_trace_source:
      ssl: false
  sink:
    - pipeline:
        name: "raw-pipeline"
raw-pipeline:
  source:
    pipeline:
      name: "entry-pipeline"
  prepper:
     - otel_trace_raw_prepper:
  sink:
    - elasticsearch:
        hosts: [ "http://<ELASTIC_HOST>:9200" ]
        trace_analytics_raw: true

OpenTelemetry Export Traces to Elastic APM and Elastic OpenDistro

1 Answers1