11

I am running an EMR cluster and trying to use a Zeppelin notebook for data analysis.

Versions: Release label:emr-5.2.1
Hadoop distribution: Amazon 2.7.3
Hive 2.1.0
Spark 2.0.2
Zeppelin 0.6.2

I am consistently having problems with Zeppelin hanging up while running a query and I can never recover it. I have tried:
- restarting the interpreter
- SSH'ing into the master node and running zeppelin_daemon.sh restart (have tried running as hadoop / root / zeppelin, and also running the script with options reload, start/stop, upstart)

Every time I use the daemon shell script, it tells me that it has stopped/started OK, but then I get this when I run a status:
Zeppelin running but process is dead [FAILED]

The only thing that I can seem to do is kill my cluster and stand up a new one, which is crazy on many levels.

So...what is the officially supported way to restart Zeppelin???

Andy Jobe
  • 311
  • 1
  • 2
  • 8

6 Answers6

19

I ssh'ed into the master node, then ran 

$ sudo stop zeppelin
zeppelin stop/waiting

$ sudo start zeppelin
zeppelin start/running, process 24434

Immediately after, refreshing Zeppelin gave me a 503, but after ~2 minutes or so, the UI was available again. I checked the logs in /var/log/zeppelin and it takes some time to configure settings.

soaptree
  • 431
  • 4
  • 6
8

Services on EMR use upstart, and the supported way to restart them is to use "sudo stop <service-name>; sudo start <service-name>". (The start and stop commands are in /sbin, which is in the PATH by default.)

Note: currently on EMR there is a limitation that prevents /sbin/restart from working properly, so you must use stop then start instead of restart.

Jonathan Kelly
  • 1,940
  • 11
  • 14
  • 1
    I tried this `sudo /sbin/stop zeppelin` and then `sudo /sbin/start zeppelin` and now the website shows a 503 error. Problem accessing /. Reason: Service Unavailable. Prior to this I tried sending SIGTERM to the PID of the zeppelin services. I am wondering what the actual command that EMR uses to run upstart, likely more than just `start zeppelin` – Davos Sep 27 '17 at 01:50
  • 1
    The fact that you are getting a 503 error (as opposed to an error connecting at all) means that Zeppelin must be running but apparently not functioning for some reason. You can check its logs in /var/log/zeppelin for a clue. BTW, yes, EMR runs daemons with no more than just “sudo /sbin/start ”. – Jonathan Kelly Sep 27 '17 at 02:40
4

Have you tried: 

sudo /usr/lib/zeppelin/bin/zeppelin-daemon.sh stop/start?
Christian Herrera Jiménez
  • 770
  • 5
  • 4
0

This might sound a bit silly, and I am not sure why it works. However, I have had problems with issuing the daemon stop/start and having it work. But if I can get the daemon running, starting up a notebook that I know works and then changing into the problem notebook can sometimes fix a hang, at least to the point where I can recover what I have done.

ClairJulia
  • 123
  • 1
  • 1
  • 5
0

on EMR 5.30 and above, you'll need to use sudo systemctl stop and sudo systemctl start commands

https://aws.amazon.com/premiumsupport/knowledge-center/restart-service-emr/

vikash dat
  • 1,494
  • 2
  • 19
  • 37
0

sudo service zeppelin start
sudo service zeppelin stop
sudo service zeppelin restart

https://zeppelin.apache.org/docs/0.7.2/install/install.html

dazilli
  • 447
  • 4
  • 5