Google Cloud Dataproc from Google Datalab

Question

Is it possible to run code located in Google Cloud Datalab on Dataproc clusters?

The idea is to use the great interactivity and interface by Datalab to run Apache Spark code.

Were you able to get an answer to this or is datalab = dataproc+jupyter notebook — mobcdi, Jul 16 '16 at 17:05

score 3 · Answer 1 · answered Aug 12 '16 at 04:29

3

This is on our radar but not yet fully enabled as an init action for a Dataproc cluster.

Thanks. Dinesh Kulkarni Product Manager, Datalab & Machine Learning, GCP

answered Aug 12 '16 at 04:29

Dinesh

421
2
7

score 1 · Answer 2 · answered Oct 15 '17 at 00:31

Now it is possible, just create a dataproc cluster using this command:

gcloud dataproc clusters create $CLUSTERNAME \
    --project $PROJECT \
    --num-workers $WORKERS \
    --bucket $BUCKET \
    --metadata startup-script-url=gs://$BUCKET/setup/setup_env.sh,BUCKET=$BUCKET \
    --master-machine-type $VMMASTER \
    --worker-machine-type $VMWORKER \
    --initialization-actions \
         gs://dataproc-initialization-actions/datalab/datalab.sh \
    --scopes cloud-platform

To make it even easier you can use this script: https://github.com/kanjih-ciandt/script-dataproc-datalab/tree/master

Google Cloud Dataproc from Google Datalab

2 Answers2