0

I am trying to export data from OTEL collector to custom exporter but it is failing with 400.

Example Reference

https://github.com/open-telemetry/opentelemetry-java-docs/tree/main/otlp/docker

OTEL Configuration

receivers:
  otlp:
    protocols:
        grpc:

exporters:
  otlphttp:
    endpoint: "http://docker.for.mac.localhost:4318"
    tls:
      insecure: true
    headers: 
      Authorization: "Basic YWRtaW46YWRtaW4="
      X-P-Stream: "demo"
      X-P-TAG-tag1: "value1"
      X-P-META-meta1: "value1"
      Content-type: "application/json"
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: promexample
    const_labels:
      label1: value1
  logging:
    loglevel: debug

  zipkin:
    endpoint: "http://zipkin-all-in-one:9411/api/v2/spans"
    format: proto

  jaeger:
    endpoint: jaeger-all-in-one:14250
    tls:
      insecure: true

# Alternatively, use jaeger_thrift_http with the settings below. In this case
# update the list of exporters on the traces pipeline.
#
#  jaeger_thrift_http:
#    url: http://jaeger-all-in-one:14268/api/traces

processors:
  batch:

extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  telemetry:
    logs:
      level: "debug"
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, zipkin, jaeger, otlphttp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, prometheus, otlphttp]

Error in OTEL Collector Logs

2023-06-02 23:11:19 2023-06-02T22:11:19.851Z    error   exporterhelper/queued_retry.go:391      Exporting failed. The error is not retryable. Dropping data.  {"kind": "exporter", "data_type": "metrics", "name": "otlphttp", "error": "Permanent error: error exporting items, request to http://docker.for.mac.localhost:4318/v1/metrics responded with HTTP Status Code 400", "dropped_items": 4}
2023-06-02 23:11:19 go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
2023-06-02 23:11:19     go.opentelemetry.io/collector/exporter@v0.78.2/exporterhelper/queued_retry.go:391
2023-06-02 23:11:19 go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
2023-06-02 23:11:19     go.opentelemetry.io/collector/exporter@v0.78.2/exporterhelper/metrics.go:125
2023-06-02 23:11:19 go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
2023-06-02 23:11:19     go.opentelemetry.io/collector/exporter@v0.78.2/exporterhelper/queued_retry.go:195
2023-06-02 23:11:19 go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
2023-06-02 23:11:19     go.opentelemetry.io/collector/exporter@v0.78.2/exporterhelper/internal/bounded_memory_queue.go:47

Error on Exporter

[2023-06-02T22:11:21Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /v1/metrics HTTP/1.1" 400 73 "-" "OpenTelemetry Collector Contrib/0.78.0 (linux/arm64)" 0.038629

What could be the reason ? I've specified all required headers in OTLP exporter section and curl request is working fine for exporter.

Investigation in OTEL source code

As per the code https://github.com/search?q=repo%3Aopen-telemetry%2Fopentelemetry-collector+Permanent+error&type=code

// permanent is an error that will be always returned if its source
// receives the same inputs.

Edit 1

After adding debug logs on exporter side logs are

[2023-06-03T08:21:21Z INFO  parseable::handlers::http::ingest] body b"\n\x91\x06\n\x93\x01\n%\n\x0cservice.name\x12\x15\n\x13OtlpExporterExample\n \n\x16telemetry.sdk.language\x12\x06\n\x04java\n%\n\x12telemetry.sdk.name\x12\x0f\n\ropentelemetry\n!\n\x15telemetry.sdk.version\x12\x08\n\x061.26.0\x12\xa1\x03\n\x1a\n\x18io.opentelemetry.example\x12\xcc\x02\n\x0bsuper_timer\x1a\x02msJ\xb8\x02\n\xb3\x02\x11H\x18\xdc\x94g\x18e\x17\x19X\xbb'zj\x18e\x17!d\0\0\0\0\0\0\0)\0\0\0\0\x80D\xc4@2\x80\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\0\0\0\0\0\0a\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0:x\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x14@\0\0\0\0\0\0$@\0\0\0\0\0\09@\0\0\0\0\0\0I@\0\0\0\0\0\xc0R@\0\0\0\0\0\0Y@\0\0\0\0\0@o@\0\0\0\0\0@\x7f@\0\0\0\0\0p\x87@\0\0\0\0\0@\x8f@\0\0\0\0\0\x88\xa3@\0\0\0\0\0\x88\xb3@\0\0\0\0\0L\xbd@\0\0\0\0\0\x88\xc3@Y\0\0\0\0\0\0Y@a\0\0\0\0\0\0]@\x10\x02\x124\n\x0fexample_counter:!\n\x1b\x11H\x18\xdc\x94g\x18e\x17\x19X\xbb'zj\x18e\x171d\0\0\0\0\0\0\0\x10\x02\x18\x01\x12\xd4\x01\n-\n+io.opentelemetry.exporters.otlp-grpc-okhttp\x12G\n\x12otlp.exporter.seen:1\n+\x11H\x18\xdc\x94g\x18e\x17\x19X\xbb'zj\x18e\x171d\0\0\0\0\0\0\0:\x0e\n\x04type\x12\x06\n\x04span\x10\x02\x18\x01\x12Z\n\x16otlp.exporter.exported:@\n:\x11H\x18\xdc\x94g\x18e\x17\x19X\xbb'zj\x18e\x171d\0\0\0\0\0\0\0:\r\n\x07success\x12\x02\x10\x01:\x0e\n\x04type\x12\x06\n\x04span\x10\x02\x18\x01"
[2023-06-03T08:21:21Z DEBUG actix_web::middleware::logger] Error in response: SerdeError(Error("expected value", line: 2, column: 1))
[2023-06-03T08:21:21Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /v1/metrics HTTP/1.1" 400 73 "-" "OpenTelemetry Collector Contrib/0.78.0 (linux/arm64)" 0.027516

I have set compression configuration to none but still getting encoded data on exporter ?

body b"\n\x91\x06\n\x93\x01\n%\n\x0cservice.name\x12\x15\n\x13OtlpExporterExample\n \n\x16telemetry.sdk.language\x12\x06\n\x04java\n%\n\x12telemetry.sdk.name\x12\x0f\n\ropentelemetry\n!\n\x15telemetry.sdk.version\x12\x08\n\x061.26.0\x12\xd4\x01\n-\n+io.opentelemetry.exporters.otlp-grpc-okhttp\x12G\n\x12otlp.exporter.seen:1\n+\x11\xc0\xa1|\x15o\x1be\x17\x19\xe8\x1f\x89\xfbq\x1be\x171d\0\0\0\0\0\0\0:\x0e\n\x04type\x12\x06\n\x04span\x10\x02\x18\x01\x12Z\n\x16otlp.exporter.exported:@\n:\x11\xc0\xa1|\x15o\x1be\x17\x19\xe8\x1f\x89\xfbq\x1be\x171d\0\0\0\0\0\0\0:\r\n\x07success\x12\x02\x10\x01:\x0e\n\x04type\x12\x06\n\x04span\x10\x02\x18\x01\x12\xa1\x03\n\x1a\n\x18io.opentelemetry.example\x12\xcc\x02\n\x0bsuper_timer\x1a\x02msJ\xb8\x02\n\xb3\x02\x11\xc0\xa1|\x15o\x1be\x17\x19\xe8\x1f\x89\xfbq\x1be\x17!d\0\0\0\0\0\0\0)\0\0\0\0\0H\xc4@2\x80\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x05\0\0\0\0\0\0\0_\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0:x\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x14@\0\0\0\0\0\0$@\0\0\0\0\0\09@\0\0\0\0\0\0I@\0\0\0\0\0\xc0R@\0\0\0\0\0\0Y@\0\0\0\0\0@o@\0\0\0\0\0@\x7f@\0\0\0\0\0p\x87@\0\0\0\0\0@\x8f@\0\0\0\0\0\x88\xa3@\0\0\0\0\0\x88\xb3@\0\0\0\0\0L\xbd@\0\0\0\0\0\x88\xc3@Y\0\0\0\0\0\0Y@a\0\0\0\0\0@\\@\x10\x02\x124\n\x0fexample_counter:!\n\x1b\x11\xc0\xa1|\x15o\x1be\x17\x19\xe8\x1f\x89\xfbq\x1be\x171d\0\0\0\0\0\0\0\x10\x02\x18\x01"
SRJ
  • 2,092
  • 3
  • 17
  • 36

0 Answers0