spring consul services are all healthy but can't reach each other when on different hosts

Question

Running Consul and spring-cloud-consul I have five micro services all registered and showing as healthy. Each of these services is running on a different EC2 instance.

Using the name of the micro service that is registered in consul I can't get a connection through to any of them using a RestTemplate

If Service A was to call Service B with the following code

private final static String SERVICE = "http://instance-service";

@LoadBalanced
@Autowired
private RestTemplate restTemplate;

@GetMapping("/instances")
public ResponseEntity<String> instances() {
    ParameterizedTypeReference<String> reference = new ParameterizedTypeReference<String>() {
    };

    return restTemplate.exchange(SERVICE, HttpMethod.GET, null, reference);
}

Service A throws

2017-12-19 17:30:36.000 ERROR 1 --- [nio-8443-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://instance-service": Connection refused (Connection refused); nested exception is java.net.ConnectException: Connection refused (Connection refused)] with root cause

and even with debugging set to TRACE I can see nothing coming in at all on Service B.

However if I take my identical setup and run it on a single EC2 instance everything works as expected.

All services are running in docker, but the consul agent is running on the host of each EC2 instance.

Each micro service has the following bootstrap.yml set

spring:
  cloud:
    consul:
      config:
        enabled: true
      discovery:
        instance-id: ${spring.application.name}
        prefer-ip-address: true

If I give the RestTemplate a random endpoint it returns Unknown host so it knows the micro service is there, it's just being blocked somewhere before it hits the codebase.

How can I debug this further to find the real reason the services are getting rejected when running on different hosts?

I have tried opening all ports on the security config just to rule that out but it made no difference.

Edit: Looking at the Consul UI - I see my services are registered under the a private IP (172.18.0.2).

Should this be a public facing address?

Gal Ben-Haim · Answer 1 · 2017-12-23T18:51:04.957

172.18.0.2 is an ip in a a docker bridged network, you definitely want the ip address that is registered in consul to be accessible by clients that use the service discovery..

I don't know your network setup or needs but you can (some ideas, there are many) -

if you expose ports to the docker host (-p), make sure the ip of the docker host is registered for each service instance (I'm not familiar with bootstrap.yml or spring cloud)
run docker containers with --net=host
register the services in consul in different methods and outside of spring cloud's scope (API call to the consul agent, service definition file, https://github.com/gliderlabs/registrator, ...)

spring consul services are all healthy but can't reach each other when on different hosts

1 Answers1