Spark session could not be created from hue server

Question

I am trying to create a spark session from hue server which has been configured with livy server to communicate with kerberos enabled spark.

I am getting the below error when trying to start spark scala session using Query editor. But i can able to submit spark jar jobs from hue server and it works fine.

Stack trace:

 


     15:21:40 DEBUG Client: stopping client from cache: org.apache.hadoop.ipc.Client@70609b0c","\nYARN Diagnostics: ","Application killed by user."]}
    [23/Apr/2018 02:52:05 +0000] decorators   ERROR    Error running create_session
    Traceback (most recent call last):
      File "/home/hue/desktop/libs/notebook/src/notebook/decorators.py", line 97, in decorator
        return func(*args, **kwargs)
      File "/home/hue/desktop/libs/notebook/src/notebook/api.py", line 85, in create_session
        response['session'] = get_api(request, session).create_session(lang=session['type'], properties=properties)
      File "/home/hue/desktop/libs/notebook/src/notebook/connectors/spark_shell.py", line 227, in create_session
        raise QueryError(_('The Spark session could not be created in the cluster: %s') % info)
    QueryError

Configured livy with kerberos credentials and also it works fine while communicating with spark using curl command.

Hue Configuration:



     livy_server_host=localhost
      livy_server_port=8998 
      livy_server_session_kind=yarn
      livy_impersonation_enabled=true 
      livy_server_session_timeout=3600000

Livy Configuration:


    livy.server.csrf_protection.enabled = false
    livy.spark.master = yarn
    livy.spark.deployMode = cluster
    livy.superusers = hue,livy
    livy.impersonation.enabled = true
    livy.server.access_control.enabled = true 
    livy.server.access_control.users = *
    livy.server.auth.kerberos.keytab livy.keytab
    livy.server.auth.kerberos.principal HTTP/livyserver@FQDN
    #livy.server.auth.type kerberos
    livy.server.launch.kerberos.keytab hue.keytab
    livy.server.launch.kerberos.principal hue/hueserver@FQDN
    livy.server.session.timeout = 5h
    livy.server.request-log-retain.days = 5

Also added proxy users for both hue and livy in hadoop core-site.xml

Is this a complete exception stack? "raise QueryError(_('The Spark session could not be created in the cluster: %s') % info)" - there should be something after that.. ? Check livy and hue logs as true error is probably somewhere there. — Tagar, Apr 23 '18 at 19:57

Spark session could not be created from hue server

0 Answers0