Achieve Fault Tolerance with Consul Cluster

Question

I have created consul server cluster using different ports in localhost.

I used below commands for that.

server 1:

consul agent -server -bootstrap-expect=3 -data-dir=consul-data -ui -bind=127.0.0.1 -dns-port=8601 -http-port=8501 -serf-lan-port=8303 -serf-wan-port=8304 -server-port=8305 -node=node1

server 2:

consul agent -server -bootstrap-expect=3 -data-dir=consul-data2 -ui -bind=127.0.0.1 -dns-port=8602 -http-port=8502 -serf-lan-port=8306 -serf-wan-port=8307 -server-port=8308 -node=node2 -join=127.0.0.1:8303

server 3:

consul agent -server -bootstrap-expect=3 -data-dir=consul-data1 -ui -bind=127.0.0.1 -node=node3 -join=127.0.0.1:8303

Then I created 2 microservices using spring boot, called service_A and service_B.

Service_B calls service_A to get some data.

Both services get registered with one of the above servers.

In application.properties:

spring.cloud.consul.port=8501  #For service_A
spring.cloud.consul.port=8502  #For service_B

This works fine as Service_B discovers Service_A without any problem.

Now When I kill the consul server which service_A got registered, system fails to give results since Service_B cannot find Service_A.

How should I make this system fault tolerant, Which means even though the consul server fails, services who registered with that server automatically get registered with another server which is available in the cluster.

Further I need to know how consul achieves High availability and fault tolerance in service registration and discovery. Hope you get the Question.

Yes, how could I make my services keep connected to the consul cluster? — Ishara Madhawa, Oct 02 '18 at 10:47

score 2 · Accepted Answer · answered Oct 08 '18 at 06:02

Apparently, you can deploy a consul cluster in your local machine but you cannot expect any resilience mechanism or fault tolerance in that same local machine. It's because your spring services (service_A & service_B) has been configured to identify the consul server which runs in the given consul server port under bootstrap.yml (default 8500).

 spring:
   cloud:
    consul:
     config:
      watch:
       enabled: true
     port: 8500
     discovery:
      instanceId: ${spring.application.name}:${random.value}

So each services will discover the consul servers that runs under 8500 port (you can change it as you wish). If you are running your consul cluster in your same local machine you cannot assign the same port number (8500) to each cluster nodes that need to be identified. It will be differed in order to run under same ip address. To achieve this you will need to deploy each consul nodes under different ip addresses with the same port number 8500.

8301 is the serf LAN port that used to handle gossip in the LAN. Even this port can be the same in each nodes to maintain the cluster inter-connection.

The easiest way to achieve this is that to use a private subnet in a AWS VPC.

And then you can assign separate configurations for each subnet nodes with the same port number for each server nodes so that it can be identified by your services_A & service_B with @EnableDiscoveryClient annotation.

Achieve Fault Tolerance with Consul Cluster

1 Answers1