please help if you have any idea: I am trying to connect kerberized Hortonworks hadoop clusters livy server with jupyter, I have 401 error when connecting
Is it possible to connect sparkmagic to kerberized spark livy server? if it is then I think I have some misconfiguration in the sparkmagic config json. username and password is the technical user which runs server and have right of impersonation in the hadoop cluster(proxy user) not actual username when I login in the jupyterhub
its part of my config.json:
"kernel_python_credentials" : {
"username": "username",
"password": "password",
"url": "http://mylivy.server:8999",
"auth": "Kerberos"
}
"logging_config": {
"version": 1,
"formatters": {
"magicsFormatter": {
"format": "%(asctime)s\t%(levelname)s\t%(message)s",
"datefmt": ""
}
},
"handlers": {
"magicsHandler": {
"class": "hdijupyterutils.filehandler.MagicsFileHandler",
"formatter": "magicsFormatter",
"home_path": "~/.sparkmagic"
}
},
"loggers": {
"magicsLogger": {
"handlers": ["magicsHandler"],
"level": "DEBUG",
"propagate": 0
}
}
},
"wait_for_idle_timeout_seconds": 15,
"livy_session_startup_timeout_seconds": 600,
.................................................etc............................
this is error message when I try some "hello world" in the spark or pyspark notebook or shell in the jupyter:
print("Hello World")
The code failed because of a fatal error: Invalid status code '401' from http://mylivy.server:8999/sessions with error payload: Error 401
HTTP ERROR: 401
Problem accessing /sessions. Reason:
Authentication required
Powered by Jetty:// 9.3.24.v20180605
.Some things to try: a) Make sure Spark has enough available resources for Jupyter to create a Spark context. b) Contact your Jupyter administrator to make sure the Spark magics library is configured correctly. c) Restart the kernel.
