How to connect and load files in remote BigInsights HDFS(kerberos authentication enabled) from local pyspark program for processing?
df = sqlContext.read.parquet("hdfs://<<remote_hdfs_host>>:8020/testDirectory")
Help would be much appreciated.
Asked
Active
Viewed 187 times
0
Shiva Ram
- 61
- 1
- 4
-
From Linux or Windows? Which version of Spark? Which kind of Spark build *(e.g. "with Hadoop 2.7")*? – Samson Scharfrichter Mar 20 '17 at 11:49
-
From linux(IBM DSX Spark service), tried both spark 1.6.0 with hadoop 2.7, spark 2.0 with hadoop 2.7. It did't work? – Shiva Ram Mar 23 '17 at 07:51
-
Did you create a kerberos ticket with `kinit`? And what do you mean by _"it didn't work"_ -- show the error stack, that's the most important information you can provide! – Samson Scharfrichter Mar 23 '17 at 09:15