Difference in load balancer behavior

Question

I noticed that load balancer has different behavior pattern in certain scenario.

When using discovery with load balancer and one of the services is restarted on different port - then if services are running on Spring Boot 2.3.8-RELEASE, other service can find and communicate with restarted service, but if services are running on Spring Boot 2.7.8 it will only work if below setting is set to "true":

spring.cloud.loadbalancer.health-check.refetch-instances=true

I tried to create 3x brand new projects in first test using Spring Boot 2.3.8-RELEASE and in second test Spring Boot 2.7.8 to eliminate any specific setting or configuration or my actual project. I was thinking that maybe there is something specific in my actual project causing difference in behavior, but it turns out that vanilla plain setup has exactly this same problem.

In general my expectation was that setup using Spring Boot 2.3.8-RELEASE will have exactly this same behavior in case of restarting a service on different port as Spring Boot 2.7.8 - but that doesn't seems to be a case.

I noticed that load balancer has different behavior pattern in certain scenario.

When using discovery with load balancer and one of the services is restarted on different port - then if services are running on Spring Boot 2.3.8-RELEASE, other service can find and communicate with restarted service, but if services are running on Spring Boot 2.7.8 it will only work if below setting is set to "true":

spring.cloud.loadbalancer.health-check.refetch-instances=true

I tried to create 3x brand new projects in first test using Spring Boot 2.3.8-RELEASE and in second test Spring Boot 2.7.8 to eliminate any specific setting or configuration or my actual project. I was thinking that maybe there is something specific in my actual project causing difference in behavior, but it turns out that vanilla plain setup has exactly this same problem.

In general my expectation was that setup using Spring Boot 2.3.8-RELEASE will have exactly this same behavior in case of restarting a service on different port as Spring Boot 2.7.8 - but that doesn't seems to be a case.

Here are details of my tests.

My setup is 3x spring boot applications - all in this same version (first test with 2.3.8-RELEASE and second with 2.7.8):

1. Discovery service - Eureka Server

2. Test service 1 - "caller" - Eureka Client

3. Test service 2 - "receiver" - also Eureka Client

The difference in behavior can be reproduce following scenario:

1. Start Discovery service

2. Start "caller" application - wait for it to register in Discovery

3. Start "receiver" application - also wait for it to register in Discovery and also wait for "caller" application to fetch latest record from Discovery

4. Call GET endpoint in "caller" application which in turn calls GET endpoint in "receiver" application

5. At this stage everything works fine, "caller" can find and call "receiver"

6. Change server port in "receiver" application to different value and restart "receiver" application

7. Again, wait for "receiver" application to register in Discovery and "caller" application to fetch latest record from Discovery

At this stage there is a difference in the way how Spring Boot 2.3.8-RELEASE application stack behaves and 2.7.8

What happens is 2.3.8-RELEASE will work fine after changing port in "receiver" application but 2.7.8 will ONLY be able to communicate with "receiver" if below setting is set to "true":

spring.cloud.loadbalancer.health-check.refetch-instances=true

if it's not (and that is the default setting) the 2.7.8 will not be able to ever communicate with "receiver" unless "receiver" port will be reverted to original value and both applications restarted.

Here are some more significant elements of my configuration for Spring Boot 2.3.8-RELEASE:

Discovery service

Main class:

@SpringBootApplication
@EnableEurekaServer
public class EurekaApplication {
    public static void main(String[] args) {    
        SpringApplication.run(EurekaApplication.class, args);
    }
}

application.yml:

spring:
    application:
        name: eureka
eureka:
        client:
        registerWithEureka: false
        fetchRegistry: false

pom.xml

    <parent>
        <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.3.8.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.abednarski79</groupId>
    <artifactId>eureka-old</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>eureka</name>
    <description>Eureka service</description>
    <properties>
        <java.version>11</java.version>
    <spring-cloud.version>Hoxton.SR12</spring-cloud.version>
    </properties>
    <dependencies>
        <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
    </dependency>

Caller service

Rest template configuration:

@Configuration
public class RestTemplateConfig {
    
    @Bean
    @LoadBalanced
    @Qualifier("loadBalanced Rest Template")
    RestTemplate loadBalancedRestTemplate() { 
        return new RestTemplate();
    }

    @Bean
    @Qualifier("rest Template")
    RestTemplate restTemplate() {
        return new RestTemplate();
    }
}

Controller:

@RestController
public class CallerController
{
    private Rest Template restTemplate;

    public CallerController (@Qualifier("loadBalancedRestTemplate") RestTemplate restTemplate)
    {
        this.restTemplate= restTemplate;
    }

    @GetMapping(value = "/call")
    public String call()
    {
        HttpHeaders headers = new HttpHeaders();
        HttpEntity<String> entity = new HttpEntity<String> (headers);
        restTemplate.exchange("http://receiver/receive", HttpMethod. GET, entity, String.class);
        System.out.println("Called.");
        return "success";
    }
}

pom.xml - only difference vs discovery service is load balancer is added:

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-loadbalancer</artifactId>
    </dependency>

application.yml:

spring:
    application:
        name: caller
    cloud:
        loadbalancer:
        ribbon:
                enabled: false
eureka:
    instance:
        prefer-ip-address: true
    client:
        healthcheck:
        enabled: true
    serviceUrl:
            defaultZone: http://localhost:8001/eureka/

Receiver application

Controller:

@RestController
public class ReceiverController
{
    @GetMapping(value = "/receive")
    public String receive()
    {
        System.out.println("Received."); 
        return "success";
    }
}

application.yml:

spring:
    application:
        name: receiver
eureka:
    instance:
        prefer-ip-address: true
    client:
        healthcheck:
            enabled: true
        serviceUrl:
            defaultZone: http://localhost:8001/eureka/

And here are some more significant elements of my configuration for Spring Boot 2.7.8: Discovery service

pom.xml:

    <parent>
        <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.7.8</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.abednarski79</groupId>
    <artifactId>eureka</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>eureka</name>
    <description>Eureka service</description>
    <properties>
        <java.version>11</java.version>
        <spring-cloud.version>2021.0.5</spring-cloud.version>
    </properties>
    <dependencies>
        <dependency>
        <groupId>org.springframework.boot</groupId> 
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
    </dependency>

Caller service

pom.xml:

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.7.8</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.abednarski79</groupId>
    <artifactId>caller</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>caller</name>
    <description>Caller service</description>
    <properties>
        <java.version>11</java.version>
        <spring-cloud.version>2021.0.5</spring-cloud.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
        </dependency>

Rest template configuration:

@Configuration
@LoadBalancerClient(value = "receiver", configuration = LoadBalancerConfig.class) 
public class RestTemplateConfig {

    @Bean
    @LoadBalanced
    @Qualifier("loadBalanced RestTemplate")
    RestTemplate loadBalanced RestTemplate() {
        return new RestTemplate();
    }

    @Bean
    @Qualifier("rest Template")
    Rest Template restTemplate() {
        return new RestTemplate();
    }

and load balancer configuration:

public class LoadBalancerConfig
{
    @Bean
    ServiceInstanceListSupplier serviceInstanceListSupplier(ConfigurableApplicationContext context, @Qualifier ("restTemplate") RestTemplate restTemplate) {
    return ServiceInstanceListSupplier
        .builder()
            .withBlockingDiscoveryClient()
            .withBlockingHealthChecks(restTemplate)
            .build(context);
    }
}

application.yml:

spring:
    application:
        name: caller
    cloud:
        loadbalancer:
            health-check:
                refetch-instances: false
eureka:
    instance:
        prefer-ip-address: true
    client:
        healthcheck:
            enabled: true
    serviceUrl:
        defaultZone: http://localhost:9001/eureka/