3

After trying kubernetes on a few KVMs with kubeadm, I'd like to setup a proper auto-scalable cluster on AWS with kops and serve a few websites with it.

The mind-blowing magic of kops create cluster ... gives me a bunch of ec2 instances, makes the k8s API available at test-cluster.example.com and even configures my local ~/.kube/config so that I can kubectl apply -f any-stuff.yaml right away. This is just great!

I'm at the point when I can send my deployments to the cluster and configure the ingress rules – all this stuff is visible in the dashboard. However, at the moment it's not very clear how I can associate the nodes in my cluster with the domain names I've got.

In my small KVM k8s I simply install traefik and expose it on ports :80 and :443. Then I go to my DNS settings and add a few A records, which point to the public IP(s) of my cluster node(s). In AWS, there is a dynamic set of VMs, some of which may go down when the cluster is not under heavy load. So It feels like I need to use an external load balancer given that my traefik helm chart service exposes two random ports instead of fixed :80 and :443, but I'm not sure.

What are the options? What is their cost? What should go to DNS records in case if the domains are not controlled by AWS?

Alexander Kachkaev
  • 842
  • 15
  • 29
  • I'm not familiar with kops, but have you deployed an ingress controller? Domains should be handled by the ingress controller. – danielepolencic Jul 24 '17 at 18:39
  • I have, yes. On my non-AWS kubernetes cluster (where the nodes are "pets" with fixed IP addresses) I've got traefik as the ingress controller and it just uses 80 and 443. DNS records simply point to the public IP addresses and all works fine (until a node is dead). For kops/AWS I expect something different, because the fleet of nodes is not static any more. – Alexander Kachkaev Jul 24 '17 at 19:14

2 Answers2

2

Configuring your service as a LoadBalancer service is not sufficient for your cluster to to setup the actual loadbalancer, you need an ingress controller running like the one above.

You should add the kops nginx ingress addon: https://github.com/kubernetes/kops/tree/master/addons/ingress-nginx

In this case the nginx ingress controller on AWS will find the ingress and create an AWS ELB for it. I am not sure of the cost, but its worth it.

You can also consider Node Ports which you can access against the node's public ips and node port (be sure to add a rule to your security group)

You can also consider the new AWS ELB v2 or ALB which supports Http/2 and websockets. You can use the alb-ingress-controller https://github.com/coreos/alb-ingress-controller for this.

Finally if you want SSL (which you should) consider the kube-lego project which will automate getting SSL certs for you. https://github.com/jetstack/kube-lego

Jonathan Wickens
  • 739
  • 6
  • 13
  • My service type is LoadBalancer, and I see elb is created without any ingress controller created on my side – Toddams Jan 10 '18 at 08:40
  • i see the same thing @Toddams but I don't know if it's a good thing or a bad thing. – Randy L Aug 21 '18 at 20:03
  • Please note that `kube-lego` is depricated. Use `cert-manager` instead: https://github.com/jetstack/cert-manager/ – demisx May 11 '20 at 20:20
1

In my case I used nginx-ingress-controller. I think that setup with traefik will be the same.

1) Set traefik service type as loadBalancer.

Kubernetes will add an ELB rule.

2) Set CNAME or ALIAS in Route53 to ELB hostname.

You can use https://github.com/kubernetes-incubator/external-dns for synchronize exposed services and ingresses with Route53.

D.Shmelev
  • 41
  • 3