1

I have setup on my PC a miniconda python environment where I have installed the databricks-connect package and configured the tool with databricks-connect configure to connect to a databricks instance I want to use when developing code in the US.

I have a need to connect to a different a different databricks instance for developing code in the EU and I thought I could do this by setting up a different miniconda environment and installing databricks-connect in that environment and setting the configuration in that environment to point to the new databricks instance.

Alas, this did not work. When I look at databricks-connect configure in either miniconda environment, I see the same configuration in both which is the configuration I last configured.

My question therefore is: Is there a way to have multiple databricks-connect connections at the same time and toggle between the two without having to reconfigure each time?

Thank you for your time.

madisonsus
  • 11
  • 1

1 Answers1

2

Right now, databricks-connect relies on the central configuration file, and this causes problems. There are two approaches to workaround that:

  1. Use environment variables as described in the documentation, but they should be set somehow, plus you need to have different python environments for different versions of databricks-connect
  2. Specify parameters as spark configuration (see in the same documentation)
  3. For each DB cluster, do following:
  • separate python environment with name <name> & activate it
  • install databricks-connect into it
  • configure databricks-connect
  • move ~/.databricks-connect into ~/.databricks-connect-<name>
  • write wrapper script, that will activate python environment & symlink ~/.databricks-connect-<name> into ~/.databricks-connect (I have such script for Zsh, it could be too long to paste it here.)
Alex Ott
  • 80,552
  • 8
  • 87
  • 132