2

I'm using a shared EMR cluster with Jupyterhub installed. If my cluster is under heavy load, I get an error enter image description hereHow do I increase the timeout for a spark application from 60 seconds to something greater like 900 seconds (15 mins)?

sho
  • 439
  • 2
  • 7
  • 13

4 Answers4

2

Set the following properties value to a higher number 

# How long the rsc client will wait when attempting to connect to the Livy server
# livy.rsc.server.connect.timeout = 60s

In case of YARN as resource manager, the application goes to accepted state when resources are not available( application has not started yet).
The above value implies the amount of time Livy server can wait for resources to become available.

DaRkMaN
  • 1,014
  • 6
  • 9
1

I've found the correct file to adjust the timeout.

/etc/jupyter/conf/config.json

"livy_session_startup_timeout_seconds": 900

Now the timeout is set to 900 seconds vs 60 before.

sho
  • 439
  • 2
  • 7
  • 13
  • This answer needs more details. The `/etc/jupyter/conf/config.json` does not exist by default. What should the JSON structure look like? Should you just put it at the top of the object? – Durga Swaroop Sep 01 '21 at 18:31
1

I had issues with a similar error message when using EMR Studio and trying to add in a large archive file to my executors. What helped me was that I was able to find livy_session_startup_timeout_seconds as mentioned in the accepted answer and increased that value to 500.

In my case on my EMR clusters there is a shell script which creates the JSON files for kernels when using a service like EMR Studio.

The script is located in /emr/notebook-env/bin/kernel_launcher.sh.

Reference for inputs that can be used/updated in the kernel_launcher.sh file:

https://github.com/jupyter-incubator/sparkmagic/blob/master/sparkmagic/example_config.json

The JSON files when created get added to /emr/notebook-env/.sparkmagic/configs/<FILE>.json

rk92
  • 551
  • 4
  • 21
0

This was for an ambari cluster, but we had to tweak:

livy.server.yarn.app-lookup-timeout = 600s

Found it in the config template here.

jon
  • 41
  • 3