0

I have pyspark notebooks running in databricks. I connect to an external hive cluster using 'hive.Connection' from pyhive. I have my data in spark dataframes. My question is how do I write this data from dataframes in a new table in Hive which resides in a different cluster other than databricks?

Thanks

user1860447
  • 1,316
  • 8
  • 25
  • 46

1 Answers1

1

Every Databricks deployment has a central Hive metastore accessible by all clusters to persist table metadata. Instead of using the Databricks Hive metastore, you have the option to use an existing external Hive metastore instance.

This article describes how to set up Azure Databricks clusters to connect to existing external Apache Hive metastores. It provides information about recommended metastore setup and cluster configuration requirements, followed by instructions for configuring clusters to connect to an external metastore.

You may check out this article about Securing Access To Shared Metastore With Azure Databricks.

CHEEKATLAPRADEEP
  • 12,191
  • 1
  • 19
  • 42
  • thank you for your response. That was one one the first things that I did. I have gone through the page. The problem is that the hive metastore ports is not exposed to external sites like Databricks. So the only option I now I have is Hive Server2. The reads are very slow using 'hive.Connection' from pyhive. I am looking for an option that will speed up the reads also. – user1860447 Oct 01 '20 at 17:34