My company is using Databricks hosted in Azure Government. The default port for a cluster is 443 which I understand is HTTPS.
After configuring my environment variables and running databricks-connect test the command successfully spins up the cluster if it is in a terminated state BUT once it turns on the test fails.
Partial error message:
The port you specified is either being used already or invalid. Port: The port that Databricks Connect connects to
- The default Databricks Connect port is 15001
- Get current value: spark.conf.get("spark.databricks.service.port")
- Set via conf: spark.conf.set("spark.databricks.service.port", ")
When I manually change the cluster's port to 15001 and set the port environment variable accordingly databricks-connect test is successful.
I am no expert on network security. Is there a way I can keep the cluster on port 443 and still utilize Databricks Connect?
