Defining a RestTemplateBuilder bean causes traceId propagation across service boundaries to no longer work

Question

The following code demonstrates that by including a RestTemplateBuilder bean into a project, micrometer-tracing will not propagate traceIds correctly across service boundaries.

I am posting the sample code here as my client prohibits me from sharing code via GitHub.

Create two Spring Maven projects, demo-client and demo-server, referencing the code below, then run the two scripts to get the microservices running. Then, send a GET request to http://localhost:2048/tracer-response and you'll see in the demo-client logs that the traceIds match. Next, uncomment the RestTemplateBuilder bean in DemoClientApplication and re-run the scripts. Retest, and you'll see that the traceIds no longer match.

Please advise as to why this configuration no longer works. It previously worked with Spring Boot 2.7.7 and Spring Cloud 2021.0.5. If the intent from the Spring team is for this configuration to no longer be used and this is pointed out in the documentation, then I apologize for missing it. If it isn't pointed out, then in my opinion this should be highlighted as a warning, as it clearly negates the whole purpose behind using distributed tracing.

demo-client code

package com.example.democlient;

import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.client.RestTemplateBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;

@SpringBootApplication
@Slf4j
public class DemoClientApplication {

  public static void main(String[] args) {
    SpringApplication.run(DemoClientApplication.class, args);
  }

  @Bean
  public RestTemplate restTemplate(RestTemplateBuilder builder) {
    return builder.build();
  }

  // Uncomment the bean definition below, and traceId propagation will no longer work, meaning the traceId
  // values logged by demo-client and demo-server will not be the same value.
/*  @Bean
  public RestTemplateBuilder restTemplateBuilder() {
    return new RestTemplateBuilder();
  }*/
}

package com.example.democlient;

import io.micrometer.tracing.Tracer;
import jakarta.annotation.security.PermitAll;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;

@RestController
@Slf4j
public class DemoClientController {

  @Autowired RestTemplate restTemplate;
  @Autowired Tracer tracer;

  @PermitAll
  @GetMapping(value = "/tracer-response", produces = MediaType.APPLICATION_JSON_VALUE)
  public TracerResponse getClientResponse() {
    log.info("Getting the tracer response from demo server...");
    var clientTraceId = tracer.currentSpan().context().traceId();
    var clientSpanId = tracer.currentSpan().context().spanId();
    log.info(
        "Here's the traceId and spanId before the call: {} and {}", clientTraceId, clientSpanId);
    var serverTracerResponse =
        restTemplate
            .getForEntity("http://localhost:2049/tracer-response", TracerResponse.class)
            .getBody();
    var serverTraceId = serverTracerResponse.getTraceId();
    var serverSpanId = serverTracerResponse.getSpanId();
    log.info(
        "Here's the traceId and spanId from the server: {} and {}", serverTraceId, serverSpanId);
    log.info("Do the client and server traceIds match? {}", clientTraceId.equals(serverTraceId));
    return serverTracerResponse;
  }
}

package com.example.democlient;

import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.Data;

@JsonInclude(JsonInclude.Include.NON_NULL)
@Data
public class TracerResponse {
  private String spanId;
  private String traceId;
}

src/main/resources/application.yml

server:
  port: 2048
spring:
  application:
   name: demo-client

src/main/resources/logback-spring.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include
            resource="org/springframework/boot/logging/logback/defaults.xml"/>
    
    <springProperty scope="context" name="springAppName"
                    source="spring.application.name"/>
    <property name="LOG_PATTERN"
              value="%d{yyyy-MM-dd, America/New_York} | %d{HH:mm:ss.SSS, America/New_York} | %-16.16thread | %-5.5p | %-40.40logger{40} | %-39.39(Trace: %X{traceId}) | %-22.22(Span: %X{spanId}) | %m%n"/>

    <appender name="console"
              class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>${LOG_PATTERN}</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <logger name="com.example" level="DEBUG" additivity="false">
        <appender-ref ref="console"/>
    </logger>

    <logger name="org.springframework" level="WARN"
            additivity="false">
        <appender-ref ref="console"/>
    </logger>

    <root level="WARN">
        <appender-ref ref="console"/>
    </root>

</configuration>

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
        <parent>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-parent</artifactId>
            <version>3.0.1</version>
            <relativePath/> <!-- lookup parent from repository -->
        </parent>
    <groupId>com.example</groupId>
    <artifactId>demo-client</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>demo-client</name>
    <description>Demo project for Spring Boot</description>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.logging.log4j</groupId>
                    <artifactId>log4j-to-slf4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

demo-server code

package com.example.demoserver;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class DemoServerApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoServerApplication.class, args);
    }

}

package com.example.demoserver;

import io.micrometer.tracing.Tracer;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.MediaType;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@Slf4j
public class DemoServerController {

    @Autowired
    Tracer tracer;

    @GetMapping(value = "/tracer-response", produces = MediaType.APPLICATION_JSON_VALUE)
    public TracerResponse getClientResponse() {
        log.info("Getting the serverResponse...");
        log.info("traceId: {} spanId: {}",tracer.currentSpan().context().traceId(),tracer.currentSpan().context().spanId());
        var response = new TracerResponse();
        response.setTraceId(tracer.currentSpan().context().traceId());
        response.setSpanId(tracer.currentSpan().context().spanId());
        return response;
    }
}

package com.example.demoserver;

import com.fasterxml.jackson.annotation.JsonInclude;
import lombok.Data;

@JsonInclude(JsonInclude.Include.NON_NULL)
@Data
public class TracerResponse {
    private String traceId;
    private String spanId;
}

src/main/resources/application.yml

server:
  port: 2049
spring:
  application:
    name: demo-server

src/main/resources/logback-spring.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include
            resource="org/springframework/boot/logging/logback/defaults.xml"/>
    
    <springProperty scope="context" name="springAppName"
                    source="spring.application.name"/>
    <property name="LOG_PATTERN"
              value="%d{yyyy-MM-dd, America/New_York} | %d{HH:mm:ss.SSS, America/New_York} | %-16.16thread | %-5.5p | %-40.40logger{40} | %-39.39(Trace: %X{traceId}) | %-22.22(Span: %X{spanId}) | %m%n"/>

    <appender name="console"
              class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>${LOG_PATTERN}</pattern>
            <charset>utf8</charset>
        </encoder>
    </appender>

    <logger name="com.example" level="DEBUG" additivity="false">
        <appender-ref ref="console"/>
    </logger>

    <logger name="org.springframework" level="WARN"
            additivity="false">
        <appender-ref ref="console"/>
    </logger>

    <root level="WARN">
        <appender-ref ref="console"/>
    </root>

</configuration>

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.0.1</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>demo-server</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>demo-server</name>
    <description>Demo project for Spring Boot</description>
    <properties>
        <java.version>17</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.logging.log4j</groupId>
                    <artifactId>log4j-to-slf4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-properties-migrator</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

scripts to manage and run the microservices

kill_demo_server.sh

#! /bin/bash
set -e
set -x

ps -ef | grep 'demo-server-0.0.1' | kill -9 $(awk 'NR==1{print $2}') 2>/dev/null;

build_deploy_demos.sh

#! /bin/bash
set -e
set -x

cd ~/repos/demo-server;
mvn clean install;
nohup java -jar target/demo-server-0.0.1-SNAPSHOT.jar &
cd ~/repos/demo-client;
mvn clean install;
java -jar target/demo-client-0.0.1-SNAPSHOT.jar

score 2 · Answer 1 · answered Jan 19 '23 at 16:35

2

Not clear by your question why do you need a custom RestTemplateBuilder bean, but I believe there are some use-case anyway.

So, let's see if making that bean fully similar to what Spring Boot auto-configuration does helps to solve your requirements!

See RestTemplateAutoConfiguration:

@Bean
@Lazy
@ConditionalOnMissingBean
public RestTemplateBuilder restTemplateBuilder(RestTemplateBuilderConfigurer restTemplateBuilderConfigurer) {
    RestTemplateBuilder builder = new RestTemplateBuilder();
    return restTemplateBuilderConfigurer.configure(builder);
}

Therefore your custom bean should be similar:

  @Bean
  public RestTemplateBuilder restTemplateBuilder(RestTemplateBuilderConfigurer restTemplateBuilderConfigurer) {
       RestTemplateBuilder builder = new RestTemplateBuilder();
       return restTemplateBuilderConfigurer.configure(builder);
  }

That RestTemplateBuilderConfigurer delegates to respective customizers. At the moment I see this impls in my classpath:

MockServerRestTemplateCustomizer
ObservationRestTemplateCustomizer
TraceRestTemplateCustomizer

Where ObservationRestTemplateCustomizer is the one which instruments a RestTempalte with micrometer-tracing.

answered Jan 19 '23 at 16:35

Artem Bilan

113,505
11
91
118

I find it very odd that this approach would be needed in order for trace id propagation to work correctly. If it's not intended for Spring users to create a ```RestTemplateBuilder``` through its default constructor as I previously did (which worked in Spring Boot 2.7.7), then maybe the creation of such a bean through its default constructor should be prevented. From my perspective, this would be something the Spring team would be interested in preventing Spring users from doing or at least post warnings about such unintended consequences. – Keith Bennett Jan 19 '23 at 20:18
2

Having auto-configuration that backs off when the user defines their own bean is a pretty common pattern in Spring Boot. It can cause issues like this when it's not obvious that the auto-configuration has backed off. For your code, I would delete the `RestTemplateBuilder` `@Bean` method entirely and use the auto-configured variant instead. – Phil Webb Jan 20 '23 at 17:26
That's what I have already done. I posted this partly to help others who may run into this same issue (as I spent quite a bit of time tracking it down in our internal libraries) and also to discuss the potential for locking down the default constructor if it's never intended to be used this way outside of its auto-configuration. – Keith Bennett Jan 20 '23 at 18:38

Defining a RestTemplateBuilder bean causes traceId propagation across service boundaries to no longer work

1 Answers1

Linked