1

I have the following setup

Databricks Cluster:

8.4 ML (includes Apache Spark 3.1.2, Scala 2.12)

Client side:

databricks==0.2
databricks-cli==0.14.3
databricks-connect==8.1.10 (The latest version up to date 20210813)

When I run databricks-connect test to test the connection, I got this error message

java.lang.IllegalArgumentException: 
The cluster is running server version `dbr-8.4` but this client only supports Set(dbr-8.1). 
You can find a list of client releases at https://pypi.org/project/databricks-connect/#history, and install the right client version with `pip install -U databricks-connect==<version>`. 
For example, to install the latest 5.1 release, use `pip install -U databricks-connect==5.1.*`. 
To ignore this error and continue, set DEBUG_IGNORE_VERSION_MISMATCH=1.

According to the error message, I understand there is a compatibility issue with the client and server versions. However, I am already using the latest client.

  1. Does it mean the client doesn't add support for server version 8.4?
  2. Is there any way to circumvent this issue?
Alex Ott
  • 80,552
  • 8
  • 87
  • 132
fuyi
  • 2,573
  • 4
  • 23
  • 46

1 Answers1

1

Unfortunately, it's the latest version of databricks-connect for DBR 8.x., and versions for 8.2-8.4 aren't released. But it may work if you add DEBUG_IGNORE_VERSION_MISMATCH=1 before the databricks-connect test or other commands like, pyspark, spark-submit, etc.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132