5

I deployed Istio in a Kubernetes cluster hosted in AWS EKS. That created a Kubernetes service of type LoadBalancer named istio-ingressgateway with an external hostname of [redacted]-redacted.us-west-2.elb.amazonaws.com, and automatically provisioned an AWS ELB, type classic. That's great. (Note: in AWS console I do not see a hosted zone for this elb hostname, it doesn't look like I can configure aliases or whatnot)

I've been able to connect a gRPC client running outside the cluster, to a gRPC server running in the cluster by defining an istio gateway which opens port 80 to host:"*", and defining an istio virtual service which routes port 80 to my destination (some port of some kubernetes service). So far so good.

Now I would like to do this for a second gRPC endpoint in the cluster. As far as I know, my choices are: either route by opening a second port (say 81) on the ingress (which I'm choosing not to do for now). Or route by defining subdomains of [redacted]-redacted.us-west-2.elb.amazonaws.com, or route by implementing "virtual hosting", i.e. two services on the same domain where the client sends host:"svc1" in a header.

This is where I'm stuck. I don't know how do define subdomains to this ELB domain name, and I don't know if I can do virtual hosting with TLS, and if so, can I still use the "passthrough" tls mode of Istio so as not to TLS-terminate on the gateway.

1- Using subdomains: When on my developer machine I try getent hosts svc1.[redacted]-redacted.us-west-2.elb.amazonaws.com it does not map to an IP, whereas if remove the svc1 prefix DNS returns 3 IPs (guessing the 3 availability zones in us-west-2). So I can't prefix svc1 to the domain without some work.

Can I define subdomains svc1 and svc2 for a domain created by classic AWS ELB automatically provisioned by deploying Istio, and if so how? Can I do this with kubectl/istioctl to configure Istio, or do I need to do this with the AWS CLI? Can this be done without registering a domain, i'm fine with that user-unfriendly elb.amazonaws.com hostname?

2- Virtual Hosting: Alternatively I could add a host=svc1 in the gRPC metadata. I tried adding ctx := metadata.AppendToOutgoingContext(context.Background(), "Host", "svc1") to my Go client, that did not work.

Some guidance would be great.

Update 1, and one solution for plaintext I'm reading on List of HTTP header fields the "Host" header should not be used in HTTP2. Given this is gRPC it is HTTP2 and as such I should not add this as a custom header in my gRPC client requests. The solution when doing virtual hosting, in a gRPC client instead of sending a "host" header, you send an ":Authority" header, the value must correspond to the host specified in istio's gateway+virtualservice. This works for me. In my Go client I had to add dialOptions = append(dialOptions, grpc.WithAuthority("[my-service-1]")).

That solution won't work for TLS according to GoDoc grpc.WithAuthority. So I still need to find how to route 2 services from the same external IP and port when TLS is used.

Update 1.1: In virtual hosting, when TLS is used, the SNI can be passed in the CLIENT HELLO message, instead of being a gRPC header like in plaintext . Depending on your language the exact API is going to vary, but in Go I was able to pass the host name by setting the ServerName field in tls.Config. You can then gate on that hostname in your istio gateway configuration, and route based on that value in your istio virtualservice.

Update 2: I tried, and failed, to create the subdomain on AWS Route53. First I create a public hosted zone for domain:[redacted]-redacted.us-west-2.elb.amazonaws.com. Next, in that hosted zone I create an A record with name=svc1.[redacted]-redacted.us-west-2.elb.amazonaws.com, route=Alias to Classic Load Balancer, region=us-west-2, load balancer=dualstack.[redacted]-redacted.us-west-2.elb.amazonaws.com.

I later test the svc1.[redacted]-redacted.us-west-2.elb.amazonaws.com A record exists with dig, which does not return an answer. However dig does return 3 A records for [redacted]-redacted.us-west-2.elb.amazonaws.com (without the svc1 prefix),with the IPs of 3 load balancers.

mipnw
  • 2,135
  • 2
  • 20
  • 46
  • How exactly your configuration looks like? Could you provide your gateway/virtual service yamls? AFAIK you could do something like [this](https://stackoverflow.com/a/57678561/11977760), another way would be to use port 80 for first grpc service and port 31400 for the second one. My last idea would be to add custom port's for your grpc services like I already mentioned in your previous [question](https://stackoverflow.com/a/64043056/11977760). Let me know if that answer your question. – Jakub Sep 24 '20 at 09:40
  • My gateway yaml is trivial, it just opens port 80. My virtual service yaml defines 2 virtual services on the same gateway, first one is for host=svc1 with route to service1, second one is for host=svc2, with route to service2. This works. But it requires the client send the ":Authority=svcX" header, which can't be a solution when TLS is used. I am wondering how I'll update this to use TLS. It would be "cleaner" in my opinion to have 2 subdomains of the ELB hostname. This way I can more easily upgrade to using TLS. But I don't know how to subdomain the ELB hostname. – mipnw Sep 24 '20 at 18:19
  • I have a similar scenario, but with https. Two services accept traffic on path ``/`` and I can only use port ``443``. I solved it by using two DNS entries with subdomains ``abc.example.com`` and ``def.example.com`` pointing to the loadbalancer external ip. The gateway accepts both entries as hosts. For each subdomain I have one ``VirtualService`` that forwards to the correct service on ``/``. If you want I can post my config as answer. – Chris Sep 24 '20 at 18:47
  • Thank you @ChristophRaab. I would love to learn how to create 2 subdomains "abc" and "def" for domain "[redacted]-[redacted].us-west-2.elb.amazonaws.com", without having to create a custom domain name "example.com", or to be told that's not possible, I must create a custom domain name to define "abc" and "def" subdomains. – mipnw Sep 24 '20 at 19:03
  • Sorry, but I do not know of a way how to do this with istio only, except maybe with creating two ingress-gateways. I come from Azure, where I solved the subdomain part with a private DNS zone, to resolve the DNS within my network, but without registering a public domain. There should be something for AWS too. – Chris Sep 24 '20 at 19:15
  • Thanks. I understand these are the small differences between cloud vendors. Maybe AWS has a way to subdomain the ELB hostname, just need to find it. I won't create a second ingress because they cost, each one means an ELB with 3 IPs (one per availability zone). Plus once I have 40 microservices, I won't be renting 40 ELBs :) – mipnw Sep 24 '20 at 19:42

1 Answers1

0

Istio

Can I define subdomains svc1 and svc2 for a domain created by classic AWS ELB automatically provisioned by deploying Istio, and if so how?

You can't do it with istio, you have to configure that in the cloud, in your case you have to configure that on aws.

On istio you can only specify the hosts, which would be the subdomains configured on aws.

Virtual Services would look like this:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: mipnw-vs1
spec:
  hosts:
  - svc1.example.com
  http:
  - name: "svc1-route"
    match:
    - uri:
        prefix: /
    route:
    - destination:
        host: svc1.default.svc.cluster.local

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: mipnw-vs2
spec:
  hosts:
  - svc2.example.com
  http:
  - name: "svc2-route"
    match:
    - uri:
        prefix: /
    route:
    - destination:
        host: svc2.default.svc.cluster.local

Aws

I would love to learn how to create 2 subdomains "abc" and "def" for domain "[redacted]-[redacted].us-west-2.elb.amazonaws.com"

I'm not exactly sure that's what you're looking for, but I did some research in the aws documentation and I found that you could use Amazon Route 53 to achieve what you need.

Amazon Route 53 is a highly available and scalable cloud Domain Name System (DNS) web service. It is designed to give developers and businesses an extremely reliable and cost effective way to route end users to Internet applications by translating names like www.example.com into the numeric IP addresses like 192.0.2.1

The following informations are collected from:

If you host a website on multiple Amazon EC2 instances, you can distribute traffic to your website across the instances by using an Elastic Load Balancing (ELB) load balancer. The ELB service automatically scales the load balancer as traffic to your website changes over time. The load balancer also can monitor the health of its registered instances and route domain traffic only to healthy instances.

To route domain traffic to an ELB load balancer, use Amazon Route 53 to create an alias record that points to your load balancer. An alias record is a Route 53 extension to DNS. It's similar to a CNAME record, but you can create an alias record both for the root domain, such as example.com, and for subdomains, such as www.example.com. (You can create CNAME records only for subdomains.)

Amazon Route 53 also offers alias records, which are an Amazon Route 53-specific extension to DNS. You can create alias records to route traffic to selected AWS resources, including Amazon Elastic Load Balancing load balancers, Amazon CloudFront distributions, AWS Elastic Beanstalk environments, API Gateways, VPC interface endpoints, and Amazon S3 buckets that are configured as websites. Alias record typically have a type of A or AAAA, but they work like a CNAME record. Using an alias record, you can map your record name (example.com) to the DNS name for an AWS resource(elb1234.elb.amazonaws.com). Resolvers see the A or AAAA record and the IP address of the AWS resource.

And there is Question about the subdomains

Can I use 'Alias' records with my sub-domains?

Yes. You can also use Alias records to map your sub-domains (www.example.com, pictures.example.com, etc.) to your ELB load balancers, CloudFront distributions, or S3 website buckets.

Additional resources:

Jakub
  • 8,189
  • 1
  • 17
  • 31
  • I tried creating an A record to the ELB with AWS Route53. That doesn't work. I'm describing the process in Update2 in the question. – mipnw Sep 28 '20 at 16:25
  • to be clear, the question is about whether one can create subdomains without registering a custom domain name. – mipnw Sep 29 '20 at 05:14
  • As mentioned [here](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-elb-load-balancer.html) `For example, if the name of the hosted zone is example.com and you want to use acme.example.com to route traffic to your load balancer, enter acme`. Could you try to enter just `svc1` instead of `svc1.[redacted]-redacted.us-west-2.elb.amazonaws.com`? To answer your question, as far as I understand it should work without the custom domain but i'm confused by the Prerequisites `A registered domain name.` and all the stackoverflow questions I checked had their own private domains. – Jakub Sep 29 '20 at 06:40
  • I'm starting to think it is not possible to subdomain the AWS ELB domain. The only way is to register a custom domain name, create a hosted zone for it on Route53, then create an A record for the subdomain in that hosted zone. – mipnw Sep 29 '20 at 16:40