How to access Livy on a Kerberized cluster (or disable Kerberos for Livy)?

Question

I created an AWS EMR cluster with Kerberos enabled, and I am able to authenticate Hadoop with my Kerberos keytab using the Java hadoop.conf authentication method. However, I cannot access Livy as I keep running into this error:

HTTP ERROR: 401 Problem accessing /. Reason: Authentication required

I cannot find a way online to authenticate Livy in my code, the only relevant module is requests_kerberos but it requires an existing valid TGT. I am looking for a way to be able to pass in a keytab or principal name+password to authenticate Livy.

If this is not possible, how can I disable Kerberos for Livy without affecting other components like Hadoop?

You can't and you don't have to. Any serious HTTP client supports **SPNego** i.e. Kerberos-on-HTTP, out of the box. — Samson Scharfrichter, Jan 29 '20 at 23:26
Note that for performance reasons the SPNego challenge/response should be done only once on the beginning of the session; after that there's a session cookie available. — Samson Scharfrichter, Jan 29 '20 at 23:27

score 0 · Answer 1 · edited Apr 15 '21 at 05:12

To authenticate to livy via Kerberos, you need the following pre-conditions:

The user you are authenticating to must exist in all nodes (see createlinuxusers.sh)
You need to create a kerberos principal for the user in the master node (see configurekdc.sh)

For more information, can check out AWS blog

Note that there is a bug in the cloudformation template which changes the sparkmagic config.json file.

You need to revert this section

"authenticators": {
    "Kerberos": "sparkmagic.auth.kerberos.Kerberos",
    "Kerberos": "sparkmagic.auth.customauth.Authenticator",
    "Basic_Access": "sparkmagic.auth.basic.Basic"
  }

to its original state

"authenticators": {
    "Kerberos": "sparkmagic.auth.kerberos.Kerberos",
    "None": "sparkmagic.auth.customauth.Authenticator",
    "Basic_Access": "sparkmagic.auth.basic.Basic"
  }

How to access Livy on a Kerberized cluster (or disable Kerberos for Livy)?

1 Answers1