0

I have a DataProc cluster running in GCP. I ran the Livy initialization script for it, and I can access the livy/sessions link through the gateway interface. I have the following set up for my sparkmagic config.json:

{
    "kernel_python_credentials" : {
        "auth": "None",
        "url": "https://{SERVER}.dataproc.googleusercontent.com/livy"
    },
    "should_heartbeat": true,
    "livy_server_heartbeat_timeout_seconds": 60,
    "heartbeat_refresh_seconds": 5,
    "heartbeat_retry_seconds": 1,
    "ignore_ssl_errors": false
}

I can start the kernel, but if I try to execute a cell it seems to be replying back with a login page. Is there some other parameter that I need to set to make this work?

oneextrafact
  • 159
  • 1
  • 9

1 Answers1

0

For the benefit of anyone else who comes here - I was able to get connectivity going by setting up port forwarding from the livy service to my local machine. However, I ran into a problem with being unable to actually execute jobs against the cluster. It appears that Livy is more or less defunct, and specifically the currently available releases (latest 0.7.1) were not built against Spark 3 / Scala 2.12. There's no easy way to make this work.

oneextrafact
  • 159
  • 1
  • 9