Upgrade to spring boot 3.1 breaks my otel exporter

Question

After upgrading from spring boot 2.7.10 (using sleuth) to 3.1 (micrometer-tracing) I updated my otel exporter config from

SPRING_SLEUTH_OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo.xxx.svc.cluster.local:4317

to

MANAGEMENT_OTLP_TRACING_ENDPOINT=http://tempo.xxx.svc.cluster.local:4317

The log is now showing the following error:

{
    "@timestamp": "2023-06-08T17:56:22.333359047-03:00",
    "level": "ERROR",
    "message": "Failed to export spans. The request could not be executed. Full error message: Connection reset",
    "traceId": "",
    "spanId": "",
    "logger": "io.opentelemetry.exporter.internal.okhttp.OkHttpExporter",
    "thread": "OkHttp http://tempo.observability.svc.cluster.local:4317/..."
}

Has anyone had similar issue ?

version:

spring boot: 3.1
micrometer-tracing: 1.1
opentelemetry-exporter-otlp: 1.26.0
tempo: 2.1

Do these errors happen occasionally or intermittently? If occasionally, then I recommend adding enabling retry on your exporter. Not sure how to enable it in your setup, but if using the otel autoconfigure module its OTEL_EXPERIMENTAL_EXPORTER_OTLP_RETRY_ENABLD=true. Documented here: https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk-extensions/autoconfigure#otlp-exporter-retry — JackB, Jun 08 '23 at 21:36
@JackB when I start the application, i send the first request and i directly receive this error message, it seems to be a constant. — Jonathan Chevalier, Jun 08 '23 at 21:38
That sounds like a networking issue then. Please confirm network connectivity between the container the application is running in an the OTLP receiver. E.g. ping / curl the target collector from a shell in the container. — JackB, Jun 08 '23 at 22:17
@JackB pretty strange as all my other spring boot application in the same namespace are exporting to the exact same tempo instance and working perfectly. — Jonathan Chevalier, Jun 09 '23 at 12:57

score 1 · Answer 1 · answered Jun 09 '23 at 14:02

Spring boot 3.1 OtlpAutoConfiguration uses OtlpHttpSpanExporter so we need to override the configuration with:

@Configuration
@EnableConfigurationProperties(OtlpProperties.class)
public class OtlpConfiguration {

  // OtlpAutoConfiguration use HTTP by default, we update it to use GRPC
  // https://github.com/spring-projects/spring-boot/blob/main/spring-boot-project/spring-boot-actuator-autoconfigure/src/main/java/org/springframework/boot/actuate/autoconfigure/tracing/otlp/OtlpAutoConfiguration.java
  @Bean
  public OtlpGrpcSpanExporter otlpExporter(final OtlpProperties properties) {

    final OtlpGrpcSpanExporterBuilder builder =
        OtlpGrpcSpanExporter.builder()
            .setEndpoint(properties.getEndpoint())
            .setTimeout(properties.getTimeout())
            .setCompression(String.valueOf(properties.getCompression()).toLowerCase());

    for (final Entry<String, String> header : properties.getHeaders().entrySet()) {
      builder.addHeader(header.getKey(), header.getValue());
    }

    return builder.build();
  }
}

Upgrade to spring boot 3.1 breaks my otel exporter

1 Answers1