livi server fails to connect to spark cluster with authentication enabled

Question

I'am running a 3 node Spark 2.1.1 Standalone Cluster with spark.authenticate=true and the Livy Server fails to connect to it. Here is a part from my livy.conf

livy.spark.authenticate = true
livy.spark.master = spark://HOSTNAME:25000
livy.spark.authenticate.secret = SECRET

In the log I can see that by some reason the security is declared as DISABLED and I guess that this leads to this error trace...

INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(vva); groups with view permissions: Set(); users  with modify permissions: Set(vva); groups with modify permissions: Set()
INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://x.x.xxx.xx:25000...
INFO TransportClientFactory: Successfully created connection to /x.x.xxx.xx:25000 after 9 ms (0 ms spent in bootstraps)
WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master x.x.xxx.xx:25000
org.apache.spark.SparkException: Exception thrown in awaitResult:
.........
Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: Expected SaslMessage, received something else (maybe your client does not have SASL enabled?)

PS. I've tested the same in a single node environment and seems to work fine.

Thanks.

Answer 1

Found out that some of the spark parameters cannot be passed from the livy.conf and had to add them to the spark-defaults.conf.