1

Anyone having some insights regarding achieving fault tolerance in Apache Livy. Say for instance the Livy server fails how we can achieve HA.

Sumit Khurana
  • 159
  • 1
  • 10

2 Answers2

0

Actually, using multiple Livy servers behind a load balancer doesn't work right now due to this bug: https://issues.apache.org/jira/browse/LIVY-541

In contrast, for deployments that require high availability, Livy supports session recovery using Zookeeper, which ensures that a Spark cluster remains available if the Livy server fails. After a restart, the Livy server can connect to existing sessions and roll back to the state before failing.

dassum
  • 4,727
  • 2
  • 25
  • 38
  • what do you mean by a rollback to the state before failing? specifically what is the state that you are referring to here? – user1870400 Feb 07 '20 at 10:35
0

If you want Livy sessions to persis restart then just set these properties in livy.conf

livy.server.recovery.mode = recovery
livy.server.recovery.state-store = filesystem
livy.server.recovery.state-store.url = file:///home/livy

You can use hdfs:// as well for store.

Sanjay Singh
  • 957
  • 10
  • 8