GCP networking from VM compute to GKE service in same VPC

Question

I have a compute instance VM named mysql-1 in the same account, same VPC as a GKE cluster.

I have a k8s service deployed:

~ $ kubectl get services
NAME                                           TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                                        AGE
haproxy-mysql                                  LoadBalancer   10.19.242.236   8.8.8.8     3306:32375/TCP,3307:30748/TCP,3308:30064/TCP   40m 
^ replaced the actual external ip with 8.8.8.8 here

On the VM I have mysql running on 3306

user@mysql-1:~$ netstat -ntap |grep 3306|grep LISTE
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      -
user@mysql-1:~$ ip a|grep 156
    inet 10.156.0.13/32 brd 10.156.0.13 scope global dynamic ens4

from a k8s pod I can connect to the vm:

# curl -I 10.156.0.13:3306
curl: (8) Weird server reply

from the VM I can’t connect to the haproxy service using both internal and external IPs:

user@mysql-1:~$ curl --connect-timeout 2 10.19.242.236:3306
curl: (28) Connection timed out after 2001 milliseconds
user@mysql-1:~$ curl --connect-timeout 2 8.8.8.8:3306
curl: (28) Connection timed out after 2001 milliseconds

when I use the pod ip it works :

$ kubectl get pods -A -o wide |grep haproxy-mysql
default       haproxy-mysql-6dd5f8cf64-72gdx                             1/1     Running   0          10m   10.16.1.39    gke-blabla-default-pool-370bb0fd-92rb   <none>           <none>
default       haproxy-mysql-6dd5f8cf64-zgmwb                             1/1     Running   0          10m   10.16.2.56    gke-blabla-default-pool-370bb0fd-prn7   <none>           <none>
user@mysql-1:~$ curl --connect-timeout 2 10.16.1.39:3306
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.

I have a firewall rule:

~ $ gcloud compute firewall-rules list|grep 3306
k8s-fw-ad6943bb1121311eab6fc42010a9c005  default  INGRESS    1000      tcp:3306,tcp:3307,tcp:3308          False


~ $ gcloud compute firewall-rules describe k8s-fw-ad6943bb1121311eab6fc42010a9c005
allowed:
- IPProtocol: tcp
  ports:
  - '3306'
  - '3307'
  - '3308'
creationTimestamp: '2019-11-28T11:18:23.294-08:00'
description: '{"kubernetes.io/service-name":"default/haproxy-mysql", "kubernetes.io/service-ip":"8.8.8.8"}'
direction: INGRESS
disabled: false
id: '2373636381534430096'
kind: compute#firewall
logConfig:
  enable: false
name: k8s-fw-ad6943bb1121311eab6fc42010a9c005
network: https://www.googleapis.com/compute/v1/projects/blalala-1234lab/global/networks/default
priority: 1000
selfLink: https://www.googleapis.com/compute/v1/projects/blalala-1234/global/firewalls/k8s-fw-ad6943bb1121311eab6fc42010a9c005
sourceRanges:
- 8.8.8.8/32
- 8.8.8.8/32
- 8.8.8.8/32
- 10.0.0.0/8
- 8.8.8.8/32
- 8.8.8.8/32
- 8.8.8.8/32
- 8.8.8.8/32
targetTags:
- gke-blablabla-4125f0a3-node

from a server that is explicitly listed in the block above (masked as a 8.8.8.8/32) I can connect to the service EXTERNAL-IP

What I am missing here ?

Answer 1

have you follow all the steps here Setting up HTTP Load Balancing with Ingressto set up your GKE cluster?

From what I understand, you need to Expose your Deployment as a Service internally

Answer 2

Pod IPs are routable, service IPs are not. Creating a VPC-native cluster:

Cluster IPs for internal Services are available only from within the cluster. If you want to access a Kubernetes Service from within the same VPC and region, but from outside of the cluster (for example, from a Compute Engine instance), use an internal load balancer.

Internal load balancer docs

For the external IP access I will shoot from the hip here. Maybe such rules wouldn't work where source is private IP and target is external IP (even as it is expressed as a tag there). For incoming connection that private IP wouldn't be seen as the source IP and rule wouldn't match?