microservices & service discovery with random ports

Question

My question is related to microservices & service discovery of a service which is spread between several hosts.

The setup is as follows:

• 2 docker hosts (host A & host B)
• a Consul server (service discovery)

Let’s say that I have 2 services:

• service A
• service B

Service B is deployed 10 times (with random ports): 5 times on host A and 5 times on host B.

When service A communicates with service B, for example, it sends a request to serviceB.example.com (hard coded).

In order to get an IP and a port, service A should query the Consul server for an SRV record.

It will get 10 ip:port pairs, for which the client should apply some load-balancing logic.

• Is there a simpler way to handle this without me developing a client resolver (+LB) library for that matter ?
• Is there anything like that already implemented somewhere ?
• Am I doing it all wrong ?

Answer 1

There are a few options:

• Load balance on client as you suggest for which you'll either need to find a ready-build service discovery library that works with SRV records and handles load balancing and circuit breaking. Another answer suggested Netflix' ribbon which I have not used and will only be interesting if you are on JVM. Note that if you are building your own, you might find it simpler to just use Consul's HTTP API for discovering services than DNS SRV records. That way you can "watch" for changes too rather than caching the list and letting it get stale.
• If you don't want to reinvent that particular wheel, another popular and simple option is to use a HAProxy instance as the load balancer. You can integrate it with consul via consul-template which will automatically watch for new/failed instances of your services and update LB config. HAProxy then provides robust load balancing and health checking with a lot of options (http/tcp, different balancing algorithms, etc). One possible setup is to have a local HAProxy instance on each docker host and a fixed port assigned statically to each logical service (can store it in Consul KV) so you connect to localhost:1234 for service A for example and localhost:2345 for service B. Local instance means you don't pay for extra round trip to loadbalancer instance then to the actual service instance but this might not be an issue for you.

Answer 2

I suggest you to check out Kontena. It will solve this kind of problem out of the box. Every service will have an internal DNS that you can use in communication between services. Kontena has also built-in load balancer that is very easy to use making it very easy to create and scale micro services.

There are also lot's of built-in features that will help developing containerized applications, like private image registry, VPN access to running services, secrets management, stateful services etc.

Kontena is open source project and the code is visible on Github

Answer 3

If you look for a minimal setup, you can wrap the values you receive from Consul via ribbon, Netflix' client based load balancer.

You will find it as a module for Spring Cloud.

I didn't find an up-to-date standalone example, only this link to chrisgray's dropwizard-consul implementation that is using it in a Dropwizard context. But it might serve as a starting point for you.