Databricks connect does not work from intellj?

Question

I am trying to use databricks connect to run the spark job on databricks cluster from intellj .I followed the below link documentation.

https://docs.databricks.com/dev-tools/databricks-connect.html

However I could not make it work with intellj and it throws below exception

21/10/01 18:32:07 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/10/01 18:32:07 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
Exception in thread "main" java.lang.NoSuchFieldError: JAVA_9
    at org.apache.spark.storage.StorageUtils$.<init>(StorageUtils.scala:207)
    at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala)
    at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:95)
    at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:443)
    at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:384)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:432)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:262)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:291)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:495)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2834)

I could not find a workaround this as the documentation does not say anything clearly I cross checked from intellj its pointed to correct jar directory returned by (databricks-connect get-jar-dir).Any clue on this will be helpful?

Note:databricks-connect test is returning success