Connection to Hive using python and Kerberos

Question

I'm trying to connect to hive using Python. I installed all of the dependencies required (sasl, thrift_sasl, etc..)

Here is how I try to connect:

configuration = {"hive.server2.authentication.kerberos.principal" : "hive/_HOST@REALM_HOST", "hive.server2.authentication.kerberos.keytab" : "/etc/security/keytabs/hive.service.keytab"}

connection = hive.Connection(configuration = configuration, host="host", port=port, auth="KERBEROS", kerberos_service_name = "hiveserver2")

But I get this error:

Minor code may provide more information (Cannot find KDC for realm "REALM_DOMAIN")

Whay I'm missing? Does someone has an example of an pyHive connection using kerberos?

Thank you for your help.

score 1 · Accepted Answer · answered Jul 05 '18 at 13:23

Thank you @Kishore. Actually in PySpark, the code looks like this :

import pyspark
from pyspark import SparkContext
from pyspark.sql import Row
from pyspark import SparkConf
from pyspark.sql import HiveContext
from pyspark.sql import functions as F
import pyspark.sql.types as T

def connection(self):
    conf = pyspark.SparkConf()
    conf.setMaster('yarn-client')
    sc = pyspark.SparkContext(conf=conf)

    self.cursor = HiveContext(sc)

    self.cursor.setConf("hive.exec.dynamic.partition", "true")
    self.cursor.setConf("hive.exec.dynamic.partition.mode", "nonstrict")
    self.cursor.setConf("hive.warehouse.subdir.inherit.perms", "true")
    self.cursor.setConf('spark.scheduler.mode', 'FAIR')

and you can request using :

rows = self.cursor.sql("SELECT someone FROM something")
for row in rows.collect():
    print row

I'm actually running the code via the command :

spark-submit --master yarn MyProgram.py

I guess you could using basically run the python with pyspark installed like :

python MyProgram.py

but I didn't tried so I won't assure that it's working

score 0 · Answer 2 · answered Jul 04 '18 at 12:53

I don't know in pyspark, but I am using below scala code and it is working since last one year. If you can change this code in python. Replace the value of properties based on your kerberos.

System.setProperty("hive.metastore.uris", "add hive.metastore.uris url");
System.setProperty("hive.metastore.sasl.enabled", "true")
System.setProperty("hive.metastore.kerberos.keytab.file", "add keytab")
System.setProperty("hive.security.authorization.enabled", "false")
System.setProperty("hive.metastore.kerberos.principal", "replace hive.metastore.kerberos.principal value")
System.setProperty("hive.metastore.execute.setugi", "true")
val hiveContext = new HiveContext(sparkContext)

Connection to Hive using python and Kerberos

2 Answers2