Canary Deployments using Nginx - traffic not routed to the Canary server

Question

We have our services deployed in 4 application servers. We have 2 Nginx for load balancing. We are using 1 Consul server for service discovery. We dedicated one of the app server to be Canary server to test the canary deployments using the split_clients directive. Using the Nginx Template file we are controlling the traffic to the Canary VS non-Canary servers.

I have tested the service individually on all 4 servers using curl on the localhost. Then I tested the curl command on one of the Nginx server. I am getting the response but its being returned only from the non-canary app servers. There is no traffic routed to the Canary server.

the nginx.conf file is getting populated as below:

....
upstream canary_servers{
   server server4:port max_fails=3 fail_timeout=60 weight=1;
}

upstream non_canary_servers{
   server server1:port max_fails=3 fail_timeout=60 weight=1;
   server server2:port max_fails=3 fail_timeout=60 weight=1;
   server server3:port max_fails=3 fail_timeout=60 weight=1;
}

split_clients "app${remote_addr}${date_gmt}" $app_variant{
   * canary_servers;
   70% non_canary_servers;
}
.....

What can be the problem here? what am I missing?

score 1 · Answer 1 · answered May 15 '20 at 07:59

1

Try putting the asterix on the last line:

split_clients "app${remote_addr}${date_gmt}" $app_variant{
   70% non_canary_servers;
   * canary_servers;
}

answered May 15 '20 at 07:59

Pixou

1,719
13
23

For my testing purposes I did put the exact percentage, like 70% non_canary and 30% canary. When I tested the same using an existing jmeter test. I see that the % is not exactly routed as I defined. sometimes it 71% - 75% to non_canary and rest to canary. Will the nginx exactly route 70% to non_canary and 30% to canary? – user3379502 May 26 '20 at 20:31
no it can't be exact. Let's say you have just 8 queries. You can't split that in 70%-30% (because 0.7*8 is not an integer). Even if you have a number of queries which is a multiple of 10, it's has to be on a client basis, so pretty hard to do. – Pixou May 29 '20 at 07:56

score 0 · Answer 2 · answered May 27 '20 at 14:56

0

The above problem is solved by using * or the actual %.

answered May 27 '20 at 14:56

user3379502

223
4
22

Canary Deployments using Nginx - traffic not routed to the Canary server

2 Answers2