Propagation of OpenTelemetry traces in Smallrye Kafka fails when using CloudEvents

Question

In our Quarkus based microservices using Smallrye Kafka for event streaming, we have implemented tracing using OpenTracing. Now, we're trying to propagate the tracing context between these services using io.smallrye.reactive.messaging.ce.OutgoingCloudEventMetadataBuilder:

import io.smallrye.reactive.messaging.TracingMetadata;
import io.smallrye.reactive.messaging.ce.OutgoingCloudEventMetadata;
import io.smallrye.reactive.messaging.ce.OutgoingCloudEventMetadataBuilder;
import java.net.URI;
import org.eclipse.microprofile.reactive.messaging.Message;
import org.eclipse.microprofile.reactive.messaging.Metadata;

public class CloudEventBuilder<T> {

  private static final String DATA_CONTENT_TYPE = "application/json";
  private TracingMetadata tracingMetadata;
  private T payload;

  private final OutgoingCloudEventMetadataBuilder<T> metadataBuilder =
      OutgoingCloudEventMetadata.builder();

  public CloudEventBuilder() {
    metadataBuilder
        .withSource(URI.create("my-event-source"))
        .withType("MyEvent")
        .withDataContentType(DATA_CONTENT_TYPE);
  }

  public CloudEventBuilder<T> withId(String id) {
    metadataBuilder.withId(id);
    return this;
  }

  public CloudEventBuilder<T> withPayload(T payload) {
    this.payload  = payload;
    return this;
  }

  public CloudEventBuilder<T> withTracingMetadata(TracingMetadata tracingMetadata) {
    this.tracingMetadata = tracingMetadata;
    return this;
  }


  public Message<T> build() {
    return Message.of(payload, Metadata.of(metadataBuilder.build(), tracingMetadata));
  }

}

However, our finding is that the tracing context is not being propagated in this way. On the producing side we see TracingMetadata including trace id's and spans being added to the cloud event, but in the consumer, that TracingMetadata seems to be overwritten by new trace id's and spans. Traces from the producer are lost.

Surprisingly, we found that tracing headers were propagated succesfully when omitting the io.smallrye.reactive.messaging.ce.OutgoingCloudEventMetadataBuilder. Using Messages, traces are propagated as expected and TracingMetadata had the same trace id on both sides:

import io.opentelemetry.context.Context;
import io.opentelemetry.extension.annotations.WithSpan;
import io.quarkus.opentelemetry.runtime.QuarkusContextStorage;
import io.smallrye.reactive.messaging.TracingMetadata;
import io.smallrye.reactive.messaging.kafka.IncomingKafkaRecord;
import java.util.concurrent.CompletionStage;
import javax.enterprise.context.ApplicationScoped;
import javax.inject.Inject;
import org.eclipse.microprofile.reactive.messaging.Channel;
import org.eclipse.microprofile.reactive.messaging.Emitter;
import org.eclipse.microprofile.reactive.messaging.Incoming;
import org.eclipse.microprofile.reactive.messaging.Message;
import org.jboss.logging.Logger;

@ApplicationScoped
public class KafkaService {

  @Channel("myevents")
  Emitter<MyEvent> emitter;

  @Inject TracedService tracedService;

  @WithSpan("myspan")
  public void send() {

    var myEvent = new MyEvent(45);
    var message = Message.of(myEvent);
    var msgWithMetadata = message.addMetadata(TracingMetadata.withCurrent(Context.current()));

    emitter.send(msgWithMetadata);
  }

  @WithSpan("myspan2")
  public void send2() {

    var myEvent = new MyEvent(45);
    var message = Message.of(myEvent);
    var msgWithMetadata = message.addMetadata(TracingMetadata.withCurrent(Context.current()));

    Message<MyEvent> ceEventMessage =
        new CloudEventBuilder<MyEvent>()
            .withPayload(myEvent)
            .withId("12345")
            .withTracingMetadata(TracingMetadata.withCurrent(Context.current()))
            .build();

    emitter.send(ceEventMessage);
  }
}

When sending messages using send() the tracing context is propagated as expected, but when using send2(), tracing information from the producer is lost.

Since we need to implement propagation of traces in a generic way, we want to use io.smallrye.reactive.messaging.ce.OutgoingCloudEventMetadataBuilder. Could you help us getting the propagation of tracing context to work?