Stack Overflow
Stack Overflow

Questions tagged [databricks-connect]

172 questions
3
votes
3 answers

Import notebooks in Databricks

I am using databricks-connect and VS Code to develop some python code for Databricks. I would like to code and run/test everything directly from VS Code using databricks-connect to avoid dealing with Databricks web IDE. For basic notebooks, it works…
Maxime
  • 61
  • 1
  • 5
3
votes
1 answer

How to run a non-spark code on databricks cluster?

I am able to pull the data from databricks connect and run spark jobs perfectly. My question is how to run non-spark or native python code on remote cluster. Not sharing the code due to confidentiality.
Nagarjuna Kanneganti
  • 31
  • 1
3
votes
1 answer

Using databricks-connect debugging a notebook that runs another notebook

I am able to connect to the Azure Databricks cluster from my Linux Centos VM, using visual studio code. Below code even works without any issue from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate() print("Cluster access…
Tim
  • 1,321
  • 1
  • 22
  • 47
3
votes
0 answers

No module found error while using Databricks-connect

I have custom libraries that I have installed on my cluster using EGG files and also on my local machine. I use databricks connect to connect my IDE to my cluster. When I try importing the module, it gives me error module not found when executed…
Sagar Moghe
  • 67
  • 1
  • 7
3
votes
2 answers

IllegalArgumentException: A project ID is required for this service but could not be determined from the builder or the environment

I'm trying to connect BigQuery Dataset to Databrick and run Script using Pyspark. Procedures I've done: I patched the BigQuery Json API to databrick in dbfs for connection access. Then I added spark-bigquery-latest.jar in the cluster library and I…
Naveen B
  • 113
  • 3
  • 12
3
votes
1 answer

Dockerfile can't copy specified local directory & file

Tried a lot to make it work since days but since I'm unable to had to drop a query on SO to get it clarified. So, I've following project structre for which I want to build dockerfile. As you can see from above project structure, I've 2 files about…
Debasish Kanhar
  • 1,123
  • 2
  • 15
  • 27
3
votes
1 answer

Databricks Connect with Azure Event Hubs

I'm facing issues while trying to run some Python code on Databricks using databricks-connect and depending on a Maven installed extension (in this case com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.17 found on Databricks official documentation…
dernat71
  • 365
  • 4
  • 16
3
votes
1 answer

Common metadata in databricks cluster

I have a 3-4 clusters in my databricks instance of Azure cloud platform. I want to maintain a common metastore for all the cluster. Let me know if anyone implemented this.
pankajs
  • 51
  • 1
  • 4
3
votes
0 answers

cannot view newly created delta table via DeltaTable.forPath

I created a table like this and inserted some data spark.sql(s"create table if not exists test " + "(key String," + "name String," + "address String," + "inserted_at TIMESTAMP) " + s" using delta LOCATION 's3://test/user/'") I can view the table…
Srinivas
  • 2,010
  • 7
  • 26
  • 51
3
votes
1 answer

Databricks Connect: DependencyCheckWarning: The java class may not be present on the remote cluster

I was performing yet another execution of local Scala code against the remote Spark cluster on Databricks and got this. Exception in thread "main" com.databricks.service.DependencyCheckWarning: The java class may not be present on the…
zaxme
  • 1,065
  • 11
  • 29
2
votes
2 answers

How to get the whole cluster information in azure databricks at the runtime?

The below code was working for the older version and the version has changed the code is not working in databricks. Latest Version :12.0 (includes Apache Spark 3.3.1, Scala…
Venkatesh
  • 91
  • 1
  • 9
2
votes
0 answers

Superset failed to connect Azure Databricks

I have installed superset with docker container, logged in to superset. I wanted to connect to Azure Databricks. I have followed the below steps. I have created "requirements-local.txt" file in /superset/docker/ folder. Added…
Nagaraj
  • 31
  • 4
2
votes
1 answer

DBUtils downloading files but files not present locally

I'm trying to use DBUtils from pyspark.dbutils outside databricks, it shows me no warning or error when copying files locally, but files are not present in target folder. I can check if the file exists on DBFS with dbutils.fs.ls, and file do…
BeGreen
  • 765
  • 1
  • 13
  • 39
2
votes
1 answer

Open file on dbfs while using databricks-connect

I am using databricks-connect to connect my local PyCharm IDE to an Azure Databricks cluster. This is working completely fine until I try to access files without a Spark Context. For example dbutils.fs.put('dbfs:/tmp/test_file.txt','line_1') with…
Rodan
  • 23
  • 3
2
votes
1 answer

ValueError: Enum ErrorCode has no value defined for name '403' in mlflow.set_experiment()

I am trying to run some code to train a model, while logging my results to MLflow on Databricks. I keep getting the following error when I try to make a call to mlflow.set_experiment(), raise ValueError('Enum {} has no value defined for name…
Minura Punchihewa
  • 1,498
  • 1
  • 12
  • 35
1
2
3
…
11 12