2

I'm running fairly simple PHP websites, on Ubuntu, with AWS EC2 instances, which use *.rds.amazonaws.com domain names to connect to their respective RDS instances.

This introduces 2 issues:

  1. When the DNS cache expires, every 5 seconds, the next request adds a minimum of 12ms to my websites processing time, which isn't ideal (I try to keep within a 100ms budget).

  2. Occasionally this resolution step fails, so rather than showing an error page to my customers, my script will sleep for ~500ms, and try again.

I'm wondering if there is something, like a local caching DNS resolver, that could handle these issues more gracefully?

Perhaps it could cache the IP address for 3 seconds, then automatically try to refresh that cache, knowing that it has 2 seconds left. And if it does continue to fail, still provide the old/stale response, as that's better than no response.

Or, it could offer up the cached response immediately, even if it has expired (it's probably still correct); and if did just provide an expired response, run an update, just incase the RDS instance has just moved.

Or, could I bodge it, with a script that checks the DNS response in a loop (dig +short), and when the IP address changes, update the /etc/hosts file, using a hostname such as database-abc.

As an aside, my sites tend to get about ~4 requests every 5 seconds during the day (so cache updates are fairly frequent), but there is low activity overnight.

Craig Francis
  • 633
  • 1
  • 8
  • 23
  • `When the DNS cache expires, every 5 seconds` Why would anyone do that? That should be a DNS worst practice. A short TTL like that is begging for an outage. – Greg Askew Sep 22 '19 at 16:39
  • Hi Greg, the 5 second TTL is set by Amazon, for their RDS service. The idea is that, if one of the database servers was to fail, it can switch over to a different server by changing the DNS entry, and that would be updated in less than 5 seconds. – Craig Francis Sep 22 '19 at 16:49
  • Not surprised. Doesn't matter, it's still a worst practice. – Greg Askew Sep 23 '19 at 10:42

1 Answers1

0

As per AWS Route 53 Documentation:

Amazon Route 53 effectively connects user requests to infrastructure running in AWS – such as Amazon EC2 instances, Elastic Load Balancing load balancers, or Amazon S3 buckets – and can also be used to route users to infrastructure outside of AWS. You can use Amazon Route 53 to configure DNS health checks to route traffic to healthy endpoints or to independently monitor the health of your application and its endpoints.

For your specific scenario, here is an article

Oxymoron
  • 340
  • 3
  • 12
  • EC2 is effectively using Route 53, it’s just that any network request adds a delay (12ms or more in my case), and a chance of failure (usually happens once every couple of days)... so I’m looking at what I can do on the server to avoid those issues, probably caching in some way. – Craig Francis Sep 22 '19 at 16:46
  • You should consider raising your TTL in the zone files. Any reason why they are set so low? Is your DNS changing often? – Oxymoron Sep 22 '19 at 16:49
  • This guy - https://www.youtube.com/watch?v=4ZtFk2dtqv0 - Nill, while he is kinda far out there, explains it very well... – Oxymoron Sep 22 '19 at 16:50
  • It’s set my Amazon, and I can’t find any way of changing it... even so, it has to be fairly low incase one of the servers was to fail (it happens quite often due to a bug they have at the moment), and whatever it’s set to, eventually a lookup needs to be done, and that will always add a delay, and a chance of failure. – Craig Francis Sep 22 '19 at 16:53
  • Please [refer to the documentation](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_BestPractices.html) – Oxymoron Sep 22 '19 at 16:55
  • Here is a specific article showing how to configure Route53 for your scenario - [Learn More](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-rds-db.html) – Oxymoron Sep 22 '19 at 16:58
  • Yeah, DNS is fun, I’ve been managing DNS servers for years (BIND, dnsmasq, etc) but I’ve never had to deal with trying to avoid these issues on the client side... I’m used to TTL’s that are much longer, and with systems that are more tolerant of delay/failure. – Craig Francis Sep 22 '19 at 16:59
  • I’m sorry, but Route53 isn’t the solution, my EC2 instance (the server running PHP) currently needs to resolve a domain name, and the network request to do that is fairly slow, and occasionally fails (no matter what is at the end of that network request)... so I’m looking for something that runs on the local machine to avoid that DNS lookup at the point the IP address is needed (to avoid delays and failures). – Craig Francis Sep 22 '19 at 17:04
  • Have you tried any sort of load balancing? – Oxymoron Sep 22 '19 at 17:05
  • 1
    I don’t think load balancing helps... if you’re talking about EC2 instances, it’s still a single server that’s handing the request, and if it’s DNS lookup for the Database server fails, that still effects that request... and if you’re talking about the DNS server end, that’s more Amazons side of things (part of RDS), and that’s on the wrong end of the network connection (so still slow, and prone to errors). – Craig Francis Sep 22 '19 at 17:11