We use Route53 for our private DNS, specifically the Multivalue Answer Routing Policy, with no Health Checks for dump/lazy load balancing.
This has worked well for us, up until yesterday where we're noticing odd behaviour.
We have a record set for ro.mysql.foo.bar.com, which contains the IPs of our read-only MySQL replicas.
If you run a DIG it reflects this;
dig A ro.mysql.foo.bar.com
;; QUESTION SECTION:
;ro.mysql.foo.bar.com. IN A
;; ANSWER SECTION:
ro.mysql.foo.bar.com. 4 IN A 10.2.2.2
ro.mysql.foo.bar.com. 4 IN A 10.2.2.3
ro.mysql.foo.bar.com. 4 IN A 10.2.2.4
The expected behaviour is when mysql hits ro.mysql.foo.bar.com it connects to one of the 3 hosts at random, in a dumb/random load balanced kind of way. This worked fine up until yesterday.
If I run
mysql -h ro.mysql.foo.bar.com -e "select @@hostname;"
I'd expect the output of @@hostname to vary between 10.2.2.2, 10.2.2.3, and 10.2.2.4
Instead, it doesn't route traffic randomly across the three IPs, it will take one and only use one.
+-----------------------------------+
| @@hostname |
+-----------------------------------+
| 10.2.2.3 |
+-----------------------------------+
If we remove 10.2.2.3 from the DNS will it stop using it. If we add it back in, it'll then continue to use only 10.2.2.3 again.
We do run our own resolver via PowerDNS (which we're phasing out), but doesn't appear to be any fault there that we can see, PowerDNS has had it's DNS cache purged too.
