Stack Overflow
Stack Overflow

Questions tagged [databricks-connect]

172 questions
1
vote
1 answer

Creating database in Azure databricks on External Blob Storage giving error

I have mapped my blob storage to dbfs:/mnt/ under name /mnt/deltalake and blob storage container name is deltalake. Mounting to Dbfs is done using Azure KeyVault backed secret scope. When I try to create a database using CREATE DATABASE abc with…
user12823830
1
vote
2 answers

Configuring databricks-connect using python OS module

I want to configure databricks-connect configure through python OS module after installing databricks-connect through os.system("pip install databricks-connect==6.5") Once databricks-connect is successfully install we need to configure it by passing…
qaiser
  • 2,770
  • 2
  • 17
  • 29
1
vote
1 answer

How to fix spark.read.format("parquet") error

I'm running Scala code on Azure databricks well. Now I want to move this code from Azure notebook to eclipse. I install databricks connection following Microsoft document successfully. Pass databricks data connection test. I also installed SBT and…
user3254986
  • 89
  • 6
1
vote
0 answers

Databricks-Connect 6.6 doesn't add custom modules to Spark Context

I'm experiencing a strange behaviour in Databricks-Connect 6.6 and was wondering if anybody has seen this before and knows what causes this problem. I have created a local Spark Context via Databricks-Connect and can successfully connect to my…
Jonas
  • 61
  • 3
1
vote
1 answer

Databricks-Connect also return module not found for multiple python files job

Currently I'm connecting to databricks with local VS Code via databricks-connect. But my submmission all comes with error of module not found, which means the code in other python files not found. I tried: Move code into the folder with…
Dong Yuan
  • 11
  • 2
1
vote
0 answers

Access Network Drive in Azure Databrick Notebook

I have a network drive that stores some files. I need to access these files via my Azure Databricks notebook. Any ideas? I tried for root, dirs, files in os.walk('\\\\\\'): for name in files: print(os.path.join(root,…
Sree Nair
  • 91
  • 4
  • 11
1
vote
2 answers

Equivalent of Databricks sc.broadcast() locally with Databricks connect

I am transferring some code from Databricks notebook into Jupyter notebook locally. The following code that works in the Databricks Notebook is not working locally. res = sc.broadcast(spark.read.table(my_table)) Here is my local code: from…
Liky
  • 1,105
  • 2
  • 11
  • 32
1
vote
0 answers

SparkSession Connect to Databricks Azure

I'm using maven and scala to create a spark application that needs to connect to a cluster on azure databricks. How can i point my sparksession to connect to the databricks cluster? I saw databricks-connect, but it loads some jar files using sbt. I…
Giannis Polyzos
  • 41
  • 3
1
vote
1 answer

databricks-connect, py4j.protocol.Py4JJavaError: An error occurred while calling o342.cache

Connection to databricks works fine, working with DataFrames goes smoothly (operations like join, filter, etc). The problem appears when I call cache on a dataframe. py4j.protocol.Py4JJavaError: An error occurred while calling o342.cache. :…
kabot
  • 81
  • 2
  • 6
0
votes
0 answers

Databricks ADSL Mount with Enabled from selected virtual networks and IP addresses

I am trying to access the ADSL file from databricks. I tried using SAS Token, Access key and Also with SPark.config (Service Principle) But its still not working. storageAccountName = "adlskush" storageAccountAccessKey = "AccountKye" sasToken =…
Kushal Patil
  • 205
  • 1
  • 2
  • 14
0
votes
0 answers

Configure remote tests on Databricks with the Testing Extension

I'm currently testing the Databricks VSCode extension. I find it way easier to use for developers than dbx because of its native integration. I'd like to test its integration with the "Test Lab" extension (cf image below) I managed to make my tests…
Arthur Clerc-Gherardi
  • 495
  • 1
  • 3
  • 16
0
votes
0 answers

How to import datawarehouse to excel without using odbc?

I want to know how to import data warehouse to excel and load fast without using odbc. It's taking too long to load platform datawarehouse DATABRICKS I just tried to import the normal way via odbc, however, it's taking too long and I need to know if…
nicarcol
  • 1
0
votes
0 answers

% pip install packages in a standalone python script

I would like to install packages as I create a sparksession in a standalone python script (.py file) that will be uploaded and run in Databricks. Something like this: from pyspark.sql import SparkSession spark =…
hhp
  • 109
  • 1
  • 7
0
votes
0 answers

Installing Azure Databricks Connect

I am trying to install Azure Databricks Connect following this article: Azure Databricks installation I am running: pip3 install --upgrade "databricks-connect==13.1.*" and getting back the following error: ERROR: Could not find a version that…
user1700890
  • 7,144
  • 18
  • 87
  • 183
0
votes
1 answer

How to declare variables in Databricks SQL editor

In Databricks Notebook (Sql) I was able to declare a variable and use it also with below syntax: set name.table=(select distinct name from t1); select * from t2 where name IN ${name.table} but the same is not working in the SQL Editor. Getting…
Rita Kumari
  • 13
  • 2
1 2 3
…
11 12