I have an API that's currently served by a subdomain under our main CA e.g. api.domain.com. This API is currently served by an Nginx ingress controller and the plan is to replace that with another ingress controller (namely Ambassador) and as much as possible avoid a deployment with downtime. Since we're using AWS EKS I have the following configured:
- an ALB configured pointing to the Nginx installation
- a separate target group pointing to the Ambassador installation
- the ALB listeners configured with weighted routing to round robin between the installations (the idea is to increase/decrease traffic at will and supervise how the new Ingress Controller is managing it)
- a Route53 record set alias pointing to the Load Balancer (api.domain.com)
- ingress exposed via Nginx for the same as the record above (api.domain.com)
Since the ingress is handled by Kubernetes internal DNS what I was hoping was for the LoadBalancer to direct traffic seamlessly to both target groups (Nginx and Ambassador), but what I get is it's directing traffic only to the Ambassador target group, while the one defined also as an Nginx Ingress rule currently it just gives 503 Service Unavailable - note that the Nginx installation is reacheble through other dns mappings so everything works ok.
Any idea what I'm doing wrong? The whole idea was to do weighted routing at LB level and not DNS level to avoid DNS propagation issues.