Docker Swarm and load-balancing of Varnish persistent TCP connections (VIP)

Question

I have a Varnish HTTP cache running in front of 40 workers. These workers are running on two identical Docker nodes (node-1 and node-2, 20 on each) as a unique service (web_workers). We're using Docker Swarm with the default endpoint_mode, which is round-robin load-balancing. So Varnish forwards requests to a unique backend hostname (= Docker Swarm Virtual IP as far as I understand).

Not long ago, I noticed that nodes were receiving an unequal load of requests: node-1 processes 4 times more requests than node-2.

Varnish uses long-lived persistent TCP connections to the backend, and node-2 was started after node-1. I suspect the difference of load can be explained by the fact that node-1 was launched first, and the Docker Swarm naive round-robin load-balancing algorithm has just allocated persistent TCP connections to it before node-2 started and now it looks like this imbalance is persisting since Varnish uses long-lived TCP connections.

1. How could I confirm this theory, if it makes sense?
2. What are the possible workarounds?

I'm thinking of disabling TCP connection reuse on Varnish side, but it could result in a severe performance hit (to be tested). Another option is to not use a unique Docker service but have one per physical node. I also could restart Varnish instances as soon as the Docker service is updated, but this is a sensitive piece of infrastructure I don't want to restart. Any idea is welcome!

EDIT:

Backend configuration part of the VCL file:

backend origin {
 .between_bytes_timeout = 5s;
 .connect_timeout = 1s;
 .first_byte_timeout = 5s;
 .host = "web_origin";
 .host_header = "web_origin";
 .max_connections = 200;
 .port = "8080";
 }

sub vcl_init {
    new origin_director = directors.round_robin();
    origin_director.add_backend(origin);
}

Answer 1

The thing you need to know about DNS resolution in Varnish is that it is done at compile team. This means that when your VCL file is loaded, Varnish will resolve the hostname and stick to the IP address.

If web_origin refers to multiple IP addresses, Varnish will only choose one. When the VCL is loaded again (because of a restart or an explicit VCL reload), the other IP might be used.

If you want to do round-robin loadbalancing in VCL, you'll need at least 2 origin servers. Here's an example:

vcl 4.1;

import directors;

backend origin1 {
 .between_bytes_timeout = 5s;
 .connect_timeout = 1s;
 .first_byte_timeout = 5s;
 .host = "web_origin";
 .host_header = "web_origin1";
 .max_connections = 200;
 .port = "8080";
}

backend origin2 {
 .between_bytes_timeout = 5s;
 .connect_timeout = 1s;
 .first_byte_timeout = 5s;
 .host = "web_origin";
 .host_header = "web_origin2";
 .max_connections = 200;
 .port = "8080";
}

sub vcl_init {
    new origin_director = directors.round_robin();
    origin_director.add_backend(origin1);
    origin_director.add_backend(origin2);
}

sub vcl_recv {
    set req.backend_hint = origin_director.backend();
}

If the web_origin hostname only points to a single IP and you want to continue doing loadbalancing at the VIP level, you could consider reducing the value of the [backend_idle_timeout][1] runtime parameter.

However, if you want more control over the loadbalancing, it's best to do this in Varnish.