Stack Overflow
Stack Overflow

Questions tagged [databricks-connect]

172 questions
1
vote
0 answers

Can I use Databricks Connect with a cluster configured with port 443?

My company is using Databricks hosted in Azure Government. The default port for a cluster is 443 which I understand is HTTPS. After configuring my environment variables and running databricks-connect test the command successfully spins up the…
Amorina
  • 21
  • 1
  • 2
1
vote
1 answer

Running Scala module with databricks-connect

I've tried to follow the instructions here to set up databricks-connect with IntelliJ. My understanding is that I can run code from the IDE and it will run on the databricks cluster. I added the jar directory from the miniconda environment and moved…
steven hurwitt
  • 183
  • 2
  • 15
1
vote
0 answers

Pyspark driver is not picking up my env configurations

I am on a windows machine. Running pyspark that was set up using databricks cli. Getting this error: Python in worker has different version 3.8 than that in driver 3.9, PySpark cannot run with different minor versions. Please check environment…
jrudd
  • 113
  • 2
1
vote
1 answer

How to access DeltaLake Tables without Databrick Cluster running

I have created DeltaLake Tables on DataBricks Cluster. And I am able to access these tables from external system/application. Though I need to keep the cluster up and running all the time to be able to access the table data. Question: Is it…
AmitG
  • 519
  • 6
  • 19
1
vote
1 answer

ML Components not working in Azure Databricks (7.3.9) pointing to Azure Data Lake Store Gen2

I’m facing issues while trying to store Matplotlib graph in Azure Data Lake Store Gen2 by processing the Kmeans Elbow method from Local Pycharm pointing to Azure Databricks cluster. For the below sample piece of ML code, am getting the error. Elbow…
Mohammed Rafiq
  • 11
  • 1
  • 4
1
vote
1 answer

Can I have more than one connection in databricks-connect?

I have setup on my PC a miniconda python environment where I have installed the databricks-connect package and configured the tool with databricks-connect configure to connect to a databricks instance I want to use when developing code in the US. I…
madisonsus
  • 11
  • 1
1
vote
1 answer

Create a local sparksession when databricks is installed

I successfully installed databricks-connect and it works fine for me, but for some reasons, I want to use a local SparkSession , so just by running a SparkSeesion.builder.getOrCreate() it will use the databricks cluster not a local session in my…
Majdi
  • 73
  • 8
1
vote
0 answers

How to use PySpark delta module with Databricks-connect?

I need to read the certain version of delta file from the s3 storage and need delta package for that. When running the code with databricks-connect I'm getting the error: ModuleNotFoundError: No module named 'delta.tables' I've tried to change…
Riva Dan
  • 47
  • 6
1
vote
1 answer

How can I check if someone is using cluster with databricks connect?

When someone is connected to the Databricks cluster , I can see in Clusters details that the certain cluster is active and there are some notebooks attached. But when I'm using the cluster with databricks-connect, cluster is not running. Is there a…
Riva Dan
  • 47
  • 6
1
vote
1 answer

Except command not working in databrick sql (spark sql)

I have written this except query to get difference in record from both hive tables from databricks notebook.(I am trying to get result as we get in mssql ie only difference in resultset) select…
Avinash Singh
  • 63
  • 1
  • 12
1
vote
1 answer

Using Pyspark locally when installed using databricks-connect

I have databricks-connect 6.6.0 installed, which has a Spark version 2.4.6. I have been using the databricks cluster till now, but I am trying to switch to using a local spark session for unit testing. However, every time I run it, it still shows up…
lesk_s
  • 365
  • 1
  • 9
1
vote
1 answer

How to proper use sql/hive variables in the new databricks connect

I'm testing the new databricks connect and I often use sql variables in my python scripts on databricks, however I'm not able to use those variables through dbconnect. The example below works fine in databricks but not in dbconnect: from pyspark.sql…
vinicius.alves
  • 11
  • 1
  • 6
1
vote
2 answers

Setting Up Databricks Connect

After running databricks-connect configure, when I run databricks-connect test, I am getting "The system cannot find the path specified." and then nothing happens, no error nothing. Please help me resolve this. Since there is no error message as…
Kenneth Singh
  • 335
  • 1
  • 3
  • 15
1
vote
1 answer

How to run an image of Docker above databricksConnectDocker without error "tiny"

When I run an image above databricksConnectDocker, I've got this: tini (tini version 0.16.1 - git.0effd37) Usage: tini [OPTIONS] PROGRAM -- [ARGS] | --version Execute a program under the supervision of a valid init process (tini) Command line…
Nastasia
  • 557
  • 3
  • 22
1
vote
1 answer

Unable to execute scala code on Azure DataBricks cluster

I am trying to setup a Development environment for DataBricks, So my developers can write code using VSCODE IDE(or some other IDE) and execute the code against the DataBricks Cluster. So I went through the Documentation of DataBricks Connect and did…
Abhishek Anvekar
  • 338
  • 1
  • 5
  • 18
1 2 3
…
11 12