Stack Overflow
Stack Overflow

Questions tagged [databricks-connect]

172 questions
0
votes
1 answer

Databricks (SQL Warehouse) How to read data from postgres (jdbc)

Trying to connect database sql warehouse from postgres db but the problem I am facing is that it shows me that data source is not supported. Please see screenshot below. enter image description here
user8536590
  • 1
  • 2
0
votes
0 answers

Pyspark databricks connect library issue

I am currently coding pyspark pipelines using databricks connect library. The steps I followed are given here. This library has been installed in a virtual environment. When I try to execute this code spark.read.load(path).first() I get this…
Trijit
  • 501
  • 1
  • 4
  • 18
0
votes
1 answer

Handling spaces in the abfss using COPY INTO with Azure Databricks

I am trying to use the COPY INTO statement in Databricks to ingest CSV files from Cloud Storage. The problem is that the folder name has a space in it /AP Posted/ and when I try to refer to the path the code execution raises the below error: Error…
Arnold Souza
  • 601
  • 5
  • 16
0
votes
0 answers

Automate - data import from databricks to local using databricks CLI

I have configured databricks-CLI and its working fine for the copy to local command. databricks fs cp dbfs:/FileStore/data/filename.csv C:\Users Can i automate this in python. Using "import databricks_cli". Please let me know how to do it?
AdityaMani
  • 45
  • 1
  • 5
0
votes
0 answers

Issue while integrating PyCharm with Databricks cluster

I am trying to integrate PyCharm with my team's Databricks cluster using databricks-connect. I created a venv and did a pip install databricks-connect. Then I configured it with the necessary info of databricks cluster. When I run databricks-test, I…
Niranjan
  • 13
  • 4
0
votes
1 answer

Cannot load spark-avro jars with databricksversion 10.4

Currently, I am facing an issue since the databricks-connect runtime on our cluster was updated to 10.4. Since then, I cannot load the jars for spark-avro anymore. By Running the following code from pyspark.sql import SparkSession spark =…
Lazloo Xp
  • 858
  • 1
  • 11
  • 36
0
votes
1 answer

Error logging Spark model with mlflow to databricks registry, via databricks-connect

I'm trying to log a trained spark model on mlflow using databricks-connect. I want this model to be logged in the Databricks registry. For now, my code looks like…
itscarlayall
  • 128
  • 1
  • 14
0
votes
0 answers

`NoClassDefFoundError: spray/json/JsonWriter` error when tying to fit a synapse.ml LightGBMRegressor

I am trying to run a spark LightGBMRegressor conncting to databricks with databricks-connect using pycharm. when trying to "fit" my data I get an error NoClassDefFoundError: spray/json/JsonWriter. code i am trying to run: if…
eliavs
  • 2,306
  • 4
  • 23
  • 33
0
votes
1 answer

how to pass values to a command in Linux if it is asking yes and other parameters

I am working on connecting to a data bricks workspace from databricks-connect command using bash script i have tried the following command to configure echo "y $(databricks url) $(token) $(cluster_id) $(org_id) $(port)" | databricks-connect…
Madhu K
  • 1
0
votes
1 answer

Converting SQL Query to Databricks SQL

I have a query that I need to convert to Databricks SQL or run against a table in a Databrick environment but failing even though it works very well against tables SQL Server. The tables and query can be found here The query to convert or run in…
UpwardD
  • 739
  • 4
  • 12
  • 36
0
votes
0 answers

How do I get file metadata using Databricks Connect?

I am using Azure Databricks, which I have hooked up to a data lake and I want to get metadata such as modified date for the files in the lake. I am able to do this within Databricks itself using os.stat() as detailed in this answer, but I am…
Tim Hoare
  • 3
  • 1
0
votes
1 answer

Connect to Amazon Aurora using Pyspark

I am new in Pyspark. I am trying to connect to Amazon Aurora from Databricks using Pyspark. Can anyone please share a sample code to connect to Amazon Aurora from Databricks using Pysparks.
SK ASIF ALI
  • 85
  • 8
0
votes
1 answer

Databricks Connect: Unable to run scala program in databricks cluster from IntelliJ

Followed the steps mentioned in this docs. databricks-connect test command works fine. However, when I launch the test scala program from Intellij, I'm seeing following error: Exception in thread "main" java.lang.NoSuchMethodError:…
TeluguCoder
  • 1
  • 2
0
votes
1 answer

Databricks streaming "A file referenced in the transaction log cannot be found"

So whenever i am trying to read from a source with stream i get this error "A file referenced in the transaction log cannot be found" and it points to a file that does not exist. I have tried: Changing the checkpoint location Changing the start…
Lasse
  • 17
  • 9
0
votes
0 answers

ERROR Shell: Failed to locate the winutils binary in the hadoop binary path

I know there are many questions like this but i tried all solutions trust me. And i keep getting the same error again and again. I am trying to access spark of remote clusters and running localy by using data-bricks connect and conda env and the IDE…
Boyuis
  • 1
  • 4
1 2 3
…
11 12