Network Policy for GCP Cloud MySQL Proxy

Question

I am trying to write some Network Policies for my App, but the database connection fails as soon as I add my policy.

Here is said that the MySQL Proxy uses Ports TCP:3307 and 443 https://cloud.google.com/sql/docs/mysql/sql-proxy#how-works

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: {{ template "name" . }}
spec:
  podSelector:
    matchLabels:
      app: {{ template "name" . }}
  policyTypes:
  - Egress
  egress:
  # allow DNS resolution
  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
    - port: 443
      protocol: TCP
    - port: 3307
      protocol: TCP

EDIT: Deployment snippet:

  - name: cloudsql-proxy
    image: gcr.io/cloudsql-docker/gce-proxy:1.28.0
    command: ["/cloud_sql_proxy",
              "-instances=company-2:europe-west3:company-mysql-1=tcp:3306",
              "-verbose=false"]
    securityContext:
      readOnlyRootFilesystem: true
      allowPrivilegeEscalation: false
      privileged: false
      runAsNonRoot: true

Cloud MySQL Docs snippet:

While the Cloud SQL Auth proxy can listen on any port, it creates outgoing or egress connections to your Cloud SQL instance only on port 3307. Because Cloud SQL Auth proxy calls APIs through the domain name sqladmin.googleapis.com, which does not have a fixed IP address, all egress TCP connections on port 443 must be allowed. If your client machine has an outbound firewall policy, make sure it allows outgoing connections to port 3307 on your Cloud SQL instance's IP.

EDIT 2:

I see this now:

2022/07/22 11:12:33 error checking scopes: *url.Error Get "http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/scopes": dial tcp 169.254.169.254:80: i/o timeout | Get

Not sure what it is and allow port 80 would not be so nice I guess.

EDIT 3:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: {{ template "name" . }}
spec:
  podSelector:
    matchLabels:
      app: {{ template "name" . }}
  policyTypes:
  - Egress
  egress:
  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
    - port: 443
      protocol: TCP
    - port: 3307
      protocol: TCP
    - port: 3306
      protocol: TCP
  - to:
    - ipBlock:
        cidr: 169.254.169.254/32

I still get the error, am I doing somthing wrong?

url.Error Get "http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/scopes

Edit 4:

kubectl describe NetworkPolicy network-p-3xl2j4


Name:         network-p-3xl2j4
Namespace:    develop
Created on:   2022-07-22 14:43:04 +0200 CEST
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: network-p-3xl2j4
              meta.helm.sh/release-namespace: develop
Spec:
  PodSelector:     app=network-p-3xl2j4
  Not affecting ingress traffic
  Allowing egress traffic:
    To Port: 53/UDP
    To Port: 53/TCP
    To Port: 443/TCP
    To Port: 3307/TCP
    To Port: 3306/TCP
    To: <any> (traffic not restricted by destination)
    ----------
    To Port: <any> (traffic allowed to all ports)
    To:
      IPBlock:
        CIDR: 169.254.169.254/32
        Except:
  Policy Types: Egress

How are you starting the proxy ? If you can share the manifests it would be good. Also which kind of DB do you have postgres, mysql ? — boredabdel, Jul 22 '22 at 09:20
I added my deployment snippet: It works without the egress rules. I also tried tcp/upd 3306 & 3307. Same result. The Proxy runs as sidecar in my Pod. — Tim Schwalbe, Jul 22 '22 at 09:34
On which port the proxy itself listens ? In other terms what port does the app uses to connect to the proxy? You need to also allow that port because network policies on egress apply to localhost, even if two containers talk to each other on the same pod the policy applies. Also what error are you getting ? — boredabdel, Jul 22 '22 at 09:47
I do not know the answer but the Cloud SQL Proxy uses IAP to make connections. Check if your configuration allows traffic to/from 35.235.240.0/20. — John Hanley, Jul 22 '22 at 10:06
The proxy itself listen on port 3306 and as stated in the docs the egress connections to the Cloud MySQL is on port 3307. But it tried this and it did not work. — Tim Schwalbe, Jul 22 '22 at 10:26
Edit your question and show what you tried and the result. The error will help solve the problem. — John Hanley, Jul 22 '22 at 10:31
Based on your new edit. The IP address `169.254.169.254` is the metadata service. You should allow access to that service (actually anything at that address as you are probably breaking DNS). Details here: https://cloud.google.com/compute/docs/metadata/querying-metadata — John Hanley, Jul 22 '22 at 11:24
In your last edit, you changed a hostname. That is still visible in the edit path. If that is sensitive information, ask a moderator to delete the prior information. — John Hanley, Jul 22 '22 at 13:28
Yes I found a solution now! It was because I using workload identity. See Answer! — Tim Schwalbe, Jul 25 '22 at 11:52

Tim Schwalbe · Accepted Answer · 2022-12-02T05:19:12.730

The root of the problem was that I am using workload identity.

If you use network policy with GKE Workload Identity, you must allow egress to the following IP addresses and port numbers so your Pods can communicate with the GKE metadata server. For clusters running GKE version 1.21.0-gke.1000 and later, allow egress to 169.254.169.252/32 on port 988. For clusters running GKE versions prior to 1.21.0-gke.1000, allow egress to 127.0.0.1/32 on port 988. To avoid disruptions during auto-upgrades, allow egress to all of these IP addresses and ports.

So here is my minimal solution with all Ports needed.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: {{ template "name" . }}
spec:
  podSelector:
    matchLabels:
      app: {{ template "name" . }}
  policyTypes:
    - Egress
  egress:
    - ports:
      - port: 53
        protocol: UDP
      - port: 53
        protocol: TCP
      - port: 443
        protocol: TCP
      - port: 3307
        protocol: TCP
    - to:
      - ipBlock:
          cidr: 169.254.169.252/32
      ports:
        - protocol: TCP
          port: 988

53 TCP/UPD for DNS
443 for calling sqladmin.googleapis.com Cloud SQL Docs
988 for Workload Identity
3307 for Cloud MySQL

Network Policy for GCP Cloud MySQL Proxy

1 Answers1