0

I am trying to integrate my Scala Eclipse IDE with my Azure Databricks Cluster so that I can directly run my Spark program through Eclipse IDE on my Databricks Cluster.

I followed the official documentation of Databricks Connect(https://docs.databricks.com/dev-tools/databricks-connect.html) . I have: Installed Anaconda. Installed Python Lib 3.7 and Databricks Connect library 6.0.1. Did the Databricks Connect Configuration part(CLI part). Also, added the client libraries in the Eclipse IDE. Set the SPARK_HOME env. variable to the path which I get from running command in Anaconda, i.e. 'databricks-connect get-jar-dir'

I have not set any other environment variables apart from the one mentioned above.

Need help on finding what else is to be done to accomplish this integration, like how the ENV. variable related to connection works if running through IDE.

If someone has already done this successfully, guide me please.

user8918714
  • 21
  • 2
  • Checkout this article to integrate Eclipse IDE with Databricks: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect – CHEEKATLAPRADEEP Jan 28 '20 at 07:43
  • @CHEEKATLAPRADEEP-MSFT I have already gone through those steps in the given link as I have already mentioned. The problem is, whenever I run the Spark program from Eclipse IDE, it runs in the local desktop environment only. It is not able to connect to the databricks cluster in the first place, neither through CLI option nor through spark configuration option. – user8918714 Jan 29 '20 at 07:28

0 Answers0