2

I have set-up datalab to run on a dataproc master node using the datalab initialisation action: 

gcloud dataproc clusters create <CLUSTER_NAME> \
--initialization-actions gs://<GCS_BUCKET>/datalab/datalab.sh \
--scopes cloud-platform

This historically has worked OK. However as of 30.5 I can no longer get any code to run, however simple. I just get the "Running" progress bar. No timeouts, no error messages. How can I debug this?

anders
  • 21
  • 1
  • I've also noticed that I am getting: "Could not cross validate the widget frontend and backend versions. Error: Timeout while trying to cross validate the widget frontend and backend versions. at manager-base.js:180" – anders May 31 '17 at 06:40
  • Hey anders, i set up a datalab cluster (--num-masters=1 --master-machine-type=n1-highmem-16 --master-boot-disk-size=370GB --num-master-local-ssds=1 --num-workers=4 --num-preemptible-workers=5 --worker-machine-type=n1-highmem-16 --scopes=cloud-platform) pyspark queries like sqlContext.createDataFrame run fine and I'm able to get a pyspark dataframe. But everytime I run .count() or .show(), it just keeps saying 'running'. Any idea what might be causing this? – hkhare Jan 12 '18 at 04:55

1 Answers1

3

I just created a cluster and it seemed to work for me.

Just seeing "Running" usually means that there is not enough room in the cluster to schedule a Spark Application. Datalab loads PySpark when Python loads and that creates a YARN application. Any code will block until the YARN application is scheduled.

On the default 2 node n1-standard-4 worker cluster, with the default configs. There can only be 1 spark application. You should be able to fit two notebooks by setting --properties spark.yarn.am.memory=1g or using a larger cluster, but you will still eventually hit a limit on running notebooks per cluster.

Patrick Clay
  • 1,339
  • 7
  • 5
  • Thanks. I have recreated the cluster and recreated connections with the exact same commands as previously and this is now working. I can only suspect that it was an environmental. Thanks for the guidance on notebooks and cluster capacity. – anders Jun 01 '17 at 03:56