1

The default in-use IP address quota is only 8 but I would like to create a Dataproc cluster with more than 8 nodes. I tried to request more allocation but was rejected xD

I tried creating a cluster with my VPC network (with internal ip address only) as described in https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/network#create_a_cloud_dataproc_cluster_with_internal_ip_address_only

But the problem is that the master node cannot be SSHed into. Is there a way in which I can create a cluster with a number of nodes more than the quota, but I can also SSH into the master node?

Much appreciated! :)

Lining
  • 37
  • 6
  • Setup a VPN in your VPC. I use OpenVPN which is available in Google Marketplace. Then you can SSH to instances with private IP addresses. – John Hanley Dec 05 '18 at 02:32
  • Thank you, John Hanley. I think your approach is very good! But since the Linux script I want to run on the master node isn't that complex, I've decided to "import sys.process._" in Scala to run the Linux bash script and wrap the code in a jar, which is submitted for a Spark Job in Dataproc – Lining Dec 11 '18 at 00:38

1 Answers1

2

If you're looking for SSH access to a VM with private IPs then there are a few options detailed here.

One option is to create what it calls a "bastion" or "jump" host VM that's connected to both the private network and one that's reachable from your computer. You SSH into the bastion and then from there to the private one.

You can also use VPN as described here.

Guy Cobb
  • 46
  • 2