Sleuth does not include ParentSpan in b3 single header for message to RabbitMQ

Question

I've recently migrated from Spring Cloud Greenwich.SR2 to Hoxton.SR9, and am using Spring Sleuth to trace incoming REST calls to my Spring Boot app that trigger messages that ultimately arrive in RabbitMQ.

Obviously one of the key differences in Spring Sleuth between those two versions is that, instead of sending 3 separate message headers for trace, span and parentSpan, now only one b3 header is sent of the form {trace}-{span}-{sample}-{parentSpan} with the last two components optional.

The thing that is puzzling me is that the b3 header on the message arriving in RabbitMQ never seems to have a parentSpan component. e.g. the sequence goes something like this:

1. Within app REST controller: traceId=T, spanId=S1, parentSpanId=S0
2. Prior to sending event from within app: traceId=T, spanId=S2, parentSpanId=S1
3. Message header on RabbitMQ queue: b3=T-S2-1

In other words the message header has the right trace, but shares the span of the sending event context in the app and has no parentSpan.

Is this expected? I feel sure that with previous versions of Sleuth I would have seen a new spanId in the message header and a parentSpan that was equal to the span in the sending event context.

Have I misunderstood what is going on, or (if what I expect is sensible) is there a way to change this default behaviour.

I have the following explicit sleuth property settings:

spring.sleuth.enabled=true
spring.messaging.enabled=true
spring.supportsJoin=false

(NB: I tried the same with supportsJoin defaulted, but it didn't seem to make any difference to the lack of parentSpanId).

Debugging Information

This is what I have been able to determine from stepping through the debugger:

1. We pass a GenericMessage into AbstractMessageChannel.send().
2. Eventually the message gets passed into TracingChannelInterceptor.preSend()
3. The TracingChannelInterceptor is configured with a DeferredInjector for which setter is an instance of MessageHeaderPropagation and injectorFactory has a producerInjectorFunction=SINGLE_NO_PARENT.
4. The TracingChannelInterceptor.preSend() method calls DeferredInjector.inject(span.context(), headers), which results in the SINGLE_NO_PARENT injector adding a b3 header of the form {traceId}-{spanId}-1 to the outgoing message.

So the absence of a parentSpanId seems to originate in the fact that the TracingChannelInterceptor.injector is a DeferredInjector whose producer injector is SINGLE_NO_PARENT.

Custom Configuration

Registering the following bean to override the default propagation settings changes the behaviour to:

1. Within app REST controller: traceId=T, spanId=S1, parentSpanId=S0
2. Prior to sending event from within app: traceId=T, spanId=S2, parentSpanId=S1
3. Message header on RabbitMQ queue: b3=T-S2-1-S1

But I wonder why it's necessary to do this?

@Configuration
public class B3Configuration {
  @Bean
  Propagation.Factory customPropagationFactory() {
    return B3Propagation.newFactoryBuilder()
        .injectFormat(B3Propagation.Format.SINGLE)
        .injectFormat(Span.Kind.CLIENT, B3Propagation.Format.SINGLE)
        .injectFormat(Span.Kind.CONSUMER, B3Propagation.Format.SINGLE)
        .injectFormat(Span.Kind.PRODUCER, B3Propagation.Format.SINGLE)
        .build();
  }
}

Answer 1

It seems the default B3Propagation.Format was changed, you can see the current one in TraceBaggageConfiguration:

// Note: Versions <2.2.3 use injectFormat(MULTI) for non-remote (ex spring-messaging)
// See #1643
static final Propagation.Factory B3_FACTORY = B3Propagation.newFactoryBuilder()
        .injectFormat(B3Propagation.Format.SINGLE_NO_PARENT).build();

You can follow the details and the changes were made in spring-cloud-sleuth#1643.

If you want to change the bahavior, I would recommend creating a BaggagePropagation.FactoryBuilder instead of a Propagation.Factory since when Sleuth autoconfigures the Propagation.Factory, it does much more, see in TraceBaggageConfiguration.

E.g.:

@Bean
BaggagePropagation.FactoryBuilder baggagePropagationFactoryBuilder() {
  return BaggagePropagation.newFactoryBuilder(
          B3Propagation.newFactoryBuilder()
                  .injectFormat(B3Propagation.Format.SINGLE)
                  .build()
  );
}