0

We have a Spark master node on a Google Cloud Dataproc cluster, which we want to connect by Hostname and NOT the Internal IP. We want to connect/ping these VMs from one another. Rationale: When we drop/create any of the VMs/Clusters, the internal IP changes, but we dont want to change bunch of connection strings everytime.

GCLOUD command line lists the Master node VM:

vn524i0@m-c02zf1nylvdt ~$ gcloud compute instances list | grep anvil
anvil-dataproc-m           us-east1-a  custom-16-65536               10.22.162.40                RUNNING

From another GCP VM (in same region) when I try to ping the VM using the internal IP, I am able to (so ICMP is enabled):

vn524i0@m-c02zf1nylvdt ~$ ping 10.22.162.40 -c 1
PING 10.22.162.40 (10.22.162.40): 56 data bytes
64 bytes from 10.22.162.40: icmp_seq=0 ttl=56 time=140.232 ms

--- 10.22.162.40 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss

But when I try to ping by hostname, I get Unknown Host error:

vn524i0@m-c02zf1nylvdt ~$ ping anvil-dataproc-m
ping: cannot resolve anvil-dataproc-m: Unknown host

I've followed the global & Zonal DNS guide and used the hostname pattern suggested by Google as per this page.

zonal DNS hostname style:

vn524i0@m-c02zf1nylvdt ~$ ping anvil-dataproc-m.us-east1-a.c.PROJECT_NAME.internal -c 1
ping: cannot resolve anvil-dataproc-m.us-east1-a.c.PROJECT_NAME.internal: Unknown host

global DNS hostname style:

vn524i0@m-c02zf1nylvdt ~$ ping anvil-dataproc-m.c.PROJECT_NAME.internal -c 1
ping: cannot resolve anvil-dataproc-m.c.PROJECT_NAME.internal: Unknown host

Any guidance on how to connect / ping/nslookup by hostname and not depend upon internal IP please?

Suman Chakraborty
  • 67
  • 6
  • Since you are able to ping the private IP address, you must have a VPN or similar configured. The next step is to configure your VPN to use the VPC's DNS server. However, a quick solution is to add the hostname and IP address in your local system's **/etc/hosts** file. https://help.nexcess.net/how-to-find-the-hosts-file-on-my-mac – John Hanley Jan 14 '22 at 22:09
  • Actually we want to connect from one GCP VM to this master VM, not from Local. – Suman Chakraborty Jan 14 '22 at 22:55
  • 2
    Edit your question and clarify exactly what you are trying to do. Mentioning your Mac only confuses the question. – John Hanley Jan 14 '22 at 22:56

0 Answers0