Stack Overflow
Stack Overflow

Questions tagged [databricks-connect]

172 questions
2
votes
2 answers

databricks-connect: invalid shard address

I am trying to run some code using databricks-connect, but I am suddenly running into this error, 22/06/24 13:24:48 ERROR SparkClientManager: Fail to get the SparkClient java.util.concurrent.ExecutionException:…
Minura Punchihewa
  • 1,498
  • 1
  • 12
  • 35
2
votes
0 answers

Execute spark tests locally instead of remote

I have imported databricks jars in my project and have databricks-connect configured to my remote db cluster. When I run unit tests in Intellij, it starts running unit tests on remote machine. I want to disable this and whenever I get the spark…
Andy Smith
  • 135
  • 1
  • 8
2
votes
1 answer

Can Flink/Delta Connector create multiple databricks table

I'm using a new Flink/Databricks connector and I'm trying to understand if a single Flink sink can create multiple delta tables? Looking at the documentation the reference to the path/s3/hdfs is single location /** * Convenience method for…
Xoerk
  • 55
  • 7
2
votes
2 answers

Databricks connect test hangs indefinitely on "The system cannot find the path specified."

I have followed the instructions on this page to install databricks-connect on Windows 10, Python 3.8, databricks version 9.1, to connect to Azure Databricks…
Brendan Hill
  • 3,406
  • 4
  • 32
  • 61
2
votes
1 answer

Databricks Connect: Automatically Accept License Prompt

I am trying to write a Dockerfile that builds a container that leverages Databricks Conenect. So, I need to set-up and install Databricks Connect through Docker RUN commands. I have the following: FROM python:3.8 COPY requirements.txt /tmp/ RUN…
Paco
  • 443
  • 3
  • 10
2
votes
2 answers

Databricks FileInfo: java.lang.ClassCastException: com.databricks.backend.daemon.dbutils.FileInfo cannot be cast to com.databricks.service.FileInfo

I'm getting ClassCastException when trying to traverse the directories in mounted Databricks volume. java.lang.ClassCastException: com.databricks.backend.daemon.dbutils.FileInfo cannot be cast to com.databricks.service.FileInfo at…
marknorkin
  • 3,904
  • 10
  • 46
  • 82
2
votes
0 answers

INVALID_ARGUMENT: request failed: wildcard tables are not supported

I am trying to connect bigquery using databricks latest version(7.1+, spark 3.0) with pyspark as script editor/base language. Steps done to accomplish this: Passed bigquery API to databricks for connection Installed Spark bigquery latest jar in…
Harini
  • 21
  • 4
2
votes
2 answers

DATABRICKS connect 6.4 not able to communicate with server anymore

I am running Pycharm on my MacBook. Client settings: Python Interpreter -> Python 3.7 (dtabricks-connect-6.4) Cluster settings: Databricks Runtime Version -> 6.4 (includes Apache Spark 2.4.5, Scala 2.11) It worked well for months but suddenly,…
Zdravko
  • 35
  • 3
2
votes
3 answers

Can I use Jupyter lab to interact with databricks spark cluster using Scala?

Can I use Jupyter lab to connect to a databricks spark cluster that is hosted remotely? There are KB articles about databricks connect, which allows a scala or java client-process to control a spark cluster. Here is an…
David Beavon
  • 1,141
  • 9
  • 16
2
votes
3 answers

Databricks connect & PyCharm & remote SSH connection

Hey StackOverflowers! I run into a problem. I have set up PyCharm to be connected with an (azure) VM through SSH connection. So first i make the configuration for the ssh connection I set up the mappings I create a conda enviroment by spining up…
George Sotiropoulos
  • 1,864
  • 1
  • 22
  • 32
2
votes
3 answers

Error connecting to databricks in python with databricks-connect

I'm using databricks-connect on mac using pycharm but after I finished the configuration and tried to run databricks-connect test, I got the following error and have no idea what the problem is. I followed this documentation:…
efsee
  • 579
  • 1
  • 10
  • 22
1
vote
1 answer

Databricks-Connect: Missing sparkContext

I have issues using the newest version of databricks-connect (13.3.0). I would like to access the sparkContext and tried it as it worked for databricks-connect<13.0: from databricks.connect import DatabricksSession spark =…
Lazloo Xp
  • 858
  • 1
  • 11
  • 36
1
vote
2 answers

VSCode Extension Databricks-Connect - Use SparkSession

I am using the Databricks VSCode extension for development in an IDE. The basic functionalities are all working well. I connected to an Azure Databricks workspace with Unity Catalog enabled, selected an active cluster (DBR 13.2) and configured the…
gamezone25
  • 288
  • 2
  • 10
1
vote
1 answer

Databricks Extension for VS Code no longer displaying notebook output

After coming back from holidays a few days ago, getting back to work is always exciting. It's especially exciting when something no longer works: It appears since the update to v1.1.0 of the databricks extension for VS Code, the notebook results are…
Horst724
  • 75
  • 6
1
vote
1 answer

Unable to access databricks cluster with databricks-connect "V2" V.13.2

When trying to execute local spark code with databricks-connect 13.2.0, it does not work. I have the following issue: Error: details = "INVALID_STATE: cluster xxxxx is not Shared or Single User Cluster.…
zezzar
  • 11
  • 2
1 2
3
…
11 12