Stack Overflow
Stack Overflow

Questions tagged [databricks-connect]

172 questions
1
vote
3 answers

Switch between workspaces with databricks-connect

Is it possible to switch workspace with the use of databricks-connect? I'm currently trying to switch with: spark.conf.set('spark.driver.host', cluster_config['host']) But this gives back the following error: AnalysisException: Cannot modify the…
PvG
  • 86
  • 10
1
vote
1 answer

DB Connect and workspace notebooks returns different results

I'm using DB Connect 9.1.9. My cluster version is 9.1LTS with a single node (for test purposes). My data is stored on a S3 as a delta table. Running the following: df = spark.sql("select * from where runDate >= '2022-01-10 14:00:00' and…
Arthur Clerc-Gherardi
  • 495
  • 1
  • 3
  • 16
1
vote
1 answer

How can I build a Scala Project with Databricks Connect in Visual Studio Code?

I am currently connecting my Visual Studio Code to my Databricks Workspace using the Databricks Connect feature (local machine is Windows). To do so, I followed instructions here and here. Now, I got it to work for PySpark. Meaning that I…
DataBach
  • 1,330
  • 2
  • 16
  • 31
1
vote
0 answers

Databricks Connect : Which specified path the system cannot find?

I have the following set up in my company, and I like to use databrick on my windows machine with Pycharm I have a databrick cluster as shown below version 7.3 Java 8 Spark in C drive on this path…
Ali Saberi
  • 864
  • 1
  • 10
  • 33
1
vote
0 answers

Disconnect Virtual environment from devops in VS CODE

I created an environment in vs code but I can't install any python packages on the environment because vscode is looking up on indexes in pkgs.dev.azure (dbconnect) PS C:\Users\test\OneDrive - \Desktop\dbconnect> pip install -U…
Mr_Unchained
  • 45
  • 1
  • 8
1
vote
0 answers

Petastorm with Databricks Connect failing

Using Azure Databricks. I have petastorm==0.11.2 and databricks-connect==9.1.0 My databricks-connect session seems to be working I'm able to read in data into my remote workspace. But when I use petastorm to create a spark converter object it says…
Jamalan
  • 482
  • 4
  • 15
1
vote
2 answers

Non-interactive configuration of databricks-connect

I am setting up a development environment as a Docker container image. This will allow me and my colleagues to get up and running quickly using it as an interpreter environment. Our intended workflow is to develop code locally and execute it on an…
Wouter Hordijk
  • 123
  • 1
  • 5
1
vote
0 answers

Cannot connect Visual Studio with Azure Databricks cluster: "Missing Python executable 'python3'

I am trying to use our Azure Databricks clusters in Visual Studio running on a virtual machine. I am following the steps described here 1. Setup cluster I set up a cluster with runtime 9.1 and specify the advanced options as should. The port I set…
user3387899
  • 601
  • 5
  • 18
1
vote
1 answer

Create Spark context from Python in order to run databricks sql

I've been following this tutorial which lets me connect to Databricks from Python and then run delta table queries. However, I've stumbled upon a problem. When I run it for the FIRST time, I get the following error: Container container-name in…
anthino12
  • 770
  • 1
  • 6
  • 29
1
vote
1 answer

MLflow FileNotFound when calling spark_udf from pycharm with databricks-connect

I am saving a scikit-learn trained model to MLflow's model registry in my Windows laptop. I am using databricks-connect to connect to an Azure Databricks cluster and train models there from my local Pycharm, but for development I launch the model…
Pablo
  • 328
  • 2
  • 16
1
vote
1 answer

Where is Spark driver when you submit SparkApplication using databricks-connect?

I'm using an private server to submit some jobs on databricks using databricks-connect. Where is host Spark driver ? On private server and it just sends jobs to databricks worker ? I'm able to see .show() results directly on my server, what does it…
Gohmz
  • 1,256
  • 16
  • 31
1
vote
0 answers

Linked Service from azure Data factory to Databricks: How to parametrize?

I am using new job cluster option while creating linked service from ADF (Data factory) to Databricks with spark configs. I want to parametrize the spark config values as well as keys. I know it's quite easy to parametrize values by referring this…
Aniket Karajgikar
  • 245
  • 2
  • 13
1
vote
1 answer

How to read Azure Databricks output using API or class library

I have Azure Databrick notebook which contain SQL command. I need to capture output of SQL command and use in Dot Net core. Need help.
Prashant Rewatkar
  • 11
  • 2
1
vote
0 answers

Hide Azure Databricks logs

I am running code locally on my computer, which uses Azure Databricks cluster. Because of this I am getting a lot of "View job details at https://adb......" statements. I am logging other things, so I cannot turn off logging itself. How can I remove…
Laurynas G
  • 573
  • 9
  • 30
1
vote
1 answer

databricks-connect failed to connect to Databricks cluster runtime 8.4

I have the following setup Databricks Cluster: 8.4 ML (includes Apache Spark 3.1.2, Scala 2.12) Client side: databricks==0.2 databricks-cli==0.14.3 databricks-connect==8.1.10 (The latest version up to date 20210813) When I run databricks-connect…
fuyi
  • 2,573
  • 4
  • 23
  • 46
1 2 3
…
11 12