Stack Overflow
Stack Overflow

Questions tagged [databricks-connect]

172 questions
1
vote
0 answers

databricks-connect version 13: spark-class2.cmd not found

I install the newest version "databricks-connect==13.0.0". Now get the issue Command C:\Users\Y\AppData\Local\pypoetry\Cache\virtualenvs\X-py3.9\Lib\site-packages\pyspark\bin\spark-class2.cmd"" not found konnte nicht gefunden werden. Traceback (most…
Lazloo Xp
  • 858
  • 1
  • 11
  • 36
1
vote
1 answer

Can't import modules installed on cluster when connecting with databricks-connect

Original question I'm trying to move from databricks runtime 10.4 LTS to 11.3 LTS. I'm able to connect fine with databricks-connect, but I am not able to import the correct version of the modules that I have loaded on the cluster. I start by…
fskj
  • 874
  • 4
  • 15
1
vote
1 answer

Python in Databricks

How to even start a basic query in databricks using python? The data I need is in databricks and so far I have been using Juypterhub to pull the data and modify few things. But now I want to eliminate a step of pulling the data in Jupyterhub and…
Ziggy
  • 113
  • 6
1
vote
1 answer

Moving a Pyspark project development form Databricks UI to VSCode using databricks connect

I am inheriting a huge pyspark project and instead of using the the Databricks UI for development I would like to use vscode via databricks-connect. Because of this I am failing to determine the best practices for the following: Because the project…
tendaitakas
  • 328
  • 5
  • 18
1
vote
1 answer

Is there any way to create Databricks jobCluster through Databricks connect?

Is there any way to create Databricks jobCluster through Databricks connect? We are using All purposed cluster so far, to reduce Databricks cost we are planning to go ahead with jobCluster but unfortunately, I couldn't find a way to create…
Sandesh
  • 109
  • 7
1
vote
1 answer

Snowflake -Data-bricks : Creating snowflake temp table in data-bricks

We wanted to create snowflake temp table in databricks. Well temp table gets created but when we try to access it it says the table does note exist. We checked snowflake history and saw the temp table being created but other query for access is…
Koushur
  • 167
  • 1
  • 9
1
vote
1 answer

Practice Exam Databricks Certified Data Engineer Associate - Question 40

Why option B is not the answer for this question? Question 40 A data analyst has noticed that their Databricks SQL queries are running too slowly. They claim that this issue is affecting all of their sequentially run queries. They ask the data…
Jerry Jose James
  • 33
  • 4
1
vote
1 answer

Get cluster metric (Ganglia charts) of all clusters via REST API in Databricks

The question is specific to databricks. Is there any API to get the ganglia chart showing cluster usage? Need to get all the Ganglia charts that are available in the Databricks cluster metrics section for all the clusters via REST API calls. We are…
Scorpio
  • 511
  • 4
  • 14
1
vote
0 answers

Graphframes connectedComponents is not working if I run my spark jobs via databricks connect

Graphframe connectedComponents is throwing exceptions when i try to run my spark job from databricks connect. Here are the configurations i am using for spark session spark = ( SparkSession .builder .config( "spark.jars.packages", …
shahidammer
  • 1,026
  • 2
  • 10
  • 24
1
vote
1 answer

DbUtils: For Development environment

We have created secrets and scopes in datababricks and we are using dbUtils to access those secrets and scopes. dbUtils.getSecret(SCOPE,KEY) Unfortunately, this dbUtils function is not working on development environment, so I need advice on how to…
ConfusedDeveloper
  • 6,389
  • 4
  • 21
  • 36
1
vote
1 answer

Use remote driver with Databricks Connect

When connecting to Databricks cluster from local IDE, I take that only spark-related commands are executed in remote mode (on cluster). How about single-node operations such as scikit-learn, to_pandas. If these functions only use local machine, the…
Kenny
  • 1,902
  • 6
  • 32
  • 61
1
vote
0 answers

Databricks-connect not able to load data

After a lot of struggling i finally managed to connect to databricks remotely only to find that i'm not allowed to load for some reason. I'm posting here in the hope that someone out there has the answer. What's trying to happen here is using…
Robert Franklin
  • 160
  • 1
  • 1
  • 7
1
vote
0 answers

Databricks - Unable to run databricks-connect. Get Access Denied

I am using JRE - 1.8 and databricks-connect version 9.1.15 with python version - Python 3.7.9 openjdk version "1.8.0_41" OpenJDK Runtime Environment (build 1.8.0_41-b04) Python 3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 16:30:00) I see that…
akshat thakar
  • 1,445
  • 21
  • 29
1
vote
0 answers

Best Pyspark Testing : issue with databricks -connect

I'm currently using databricks and in order to test my databricks code I'm using databricks connect in VS code. While I'm using databricks connect, since yesterday suddenly it started behaving strange, while I'm submitting a code from VS Code, the…
dsk
  • 1,863
  • 2
  • 10
  • 13
1
vote
0 answers

How to resolve "java.lang.ClassNotFoundException" when running Scala Spark project using databricks-connect?

Currently I am learning how to use databricks-connect to develop Scala code using IDE locally. The set-up of the databricks-connect as described here was succuesful. This is an assumption as I can run PySpark Code against the cluster with no…
DataBach
  • 1,330
  • 2
  • 16
  • 31
1 2 3
…
11 12