0

We use a JDBC to connect to the hive thrift server on our spark EMR cluster. We have had some transient issues of the thrift server crashing and requiring manual restart.

We are wondering if there is any [hopefully simple] way for us to monitor the health of the thrift server? At the very least, we would like to get alerted when it goes down. It would also be nice to have other server health metrics.

James Wierzba
  • 16,176
  • 14
  • 79
  • 120

1 Answers1

1

Hive thrift server is just a spark application and its also provide you Spark UI to monitor what all queries you are executing, memory consumption and much more information.

Default port for any spark UI is 4040. But if not.

Then do execute jps command on your master machine. it will list all java application with their corresponding process ids. And fter that execute netstat -ntpul | grep <process id>, which will list all ports related to that process.

Kaushal
  • 3,237
  • 3
  • 29
  • 48
  • I will try your two suggestions. Our ultimate goal is to enable automated monitoring of thrift servers running on our (many) EMR clusters, so that we can set up alerts when there are problems. The spark UI probably wouldn't help us with this, but perhaps we can somehow configure `netstat` to dump logs and then consume them for monitoring. – James Wierzba Dec 17 '18 at 20:19