4

Background

I have a GCP project with two GKE clusters: public-cluster and private-cluster. public-cluster runs an API gateway which performs centralized authentication, logging, rate-limiting etc and reroutes requests to backend microservices running on private-cluster. The API gateway is built using Ocelot.

public-cluster is internet-facing and uses ingress-nginx to expose a public external IP address which accepts HTTP/s traffic on ports 80/443.

The intention for private-cluster is that is can only accept port 80/443 traffic from public-cluster. We do not want this cluster accessible from anywhere else, i.e. no direct HTTP/s requests to private-cluster either from within the VPC or externally, unless the traffic is from public-cluster. I have exposed the services running on private-cluster using an internal load balancer, so each service has it's own internal IP. The API gateway uses these internal IPs to reroute inbound requests to the backend microservices.

public-cluster and private-cluster are on separate subnets within the same region within the same VPC.

The intended architecture can be seen here:

in this image.

The problem

I am trying to create firewall rules which will block all traffic to the private-cluster unless it comes from the public-cluster, as follows:

  • One ingress rule with a low priority which denies all traffic to private-cluster (using the network tag as the target) and 0.0.0.0/0 as the source IP range
  • A higher priority ingress rule where:
    • Target = private-cluster
    • Source filter = public-cluster
    • Allows TCP traffic on ports 80 and 443

If I SSH onto a node within the public-cluster and fire curl requests to a service on the private-cluster (using the service's internal load balancer IP), the firewall rules above correctly allow the traffic. However, if I fire requests from my local machine to the public-cluster API Gateway, the firewall rules block the traffic. It seems like in this case the network tag in the firewall rule is being ignored.

I have tried a few things to get the rules working (all of which have been unsuccessful), such as:

  • using the subnet IP range that public-cluster sits on as the source filter IP range
  • using the subnet's gateway IP as the source filter IP
  • using public-cluster's nginx external IP as the source IP

Questions

So, my questions are:

  1. What is the correct way to define this firewall rule so that requests rerouted from the API gateway running on public-cluster are allowed through the firewall to private-cluster?
  2. More generally, is this a typical architecture pattern for Kubernetes clusters (i.e. having a public facing cluster running an API gateway which reroutes requests to a backend non-public facing cluster) and, if not, is there a better way to architect this? (I appreciate this is a very subjective question but I am interested to hear about alternative approaches)
kenlukas
  • 3,101
  • 2
  • 16
  • 26
Andrew Ridout
  • 181
  • 1
  • 5

3 Answers3

4

I got this to work by adding a firewall rule which allows ports 80/443 from public-cluster's pod address range to the private-cluster's network tags.

  1. Get public-clusters's pod address range:
gcloud container clusters describe public-cluster --zone europe-west2-a | grep clusterIpv4Cidr
  1. Create firewall rule (replace --source-ranges=XX.XX.X.X/XX with the pod address range):
gcloud compute firewall-rules create allow-public-cluster-to-private-cluster \
    --direction=INGRESS \
    --priority=1000 \
    --network=custom-vpc \
    --action=ALLOW \
    --rules=tcp:80,tcp:443 \
    --source-ranges=XX.XX.X.X/XX \
    --target-tags=private-cluster
Andrew Ridout
  • 181
  • 1
  • 5
0

The best way is to apply labels to the nodes of the public cluster and then open a port on the private cluster allowing only the given label. All the gcp firewall can be label based, non only ip or class based.

matteo nunziati
  • 664
  • 1
  • 4
  • 13
0

For vpc native clusters, the private cluster nodes will be able to talk to every other instance on that vpc, but not with any destination outside the vpc. if you want to lock down even inside that vpc, you can tag the nodes accordingly and apply firewall rules to the vm's with those tags.

Agustin E.
  • 21
  • 1