Stack Overflow
Stack Overflow

Questions tagged [azure-databricks]

For questions about the usage of Databricks Lakehouse Platform on Microsoft Azure

Overview

Azure Databricks is the Azure-based implementation of Databricks, which is a high-level platform for working with Apache Spark and includes Jupyter-style notebooks.

Azure Databricks is a first class Azure service and natively integrates with other Azure services such as Active Directory, Blob Storage, Cosmos DB, Data Lake Store, Event Hubs, HDInsight, Key Vault, Synapse Analytics, etc.

Related Tags

4095 questions
1
vote
2 answers

New databricks feature: import is not working

I have uploaded my databricks notebooks to a repo and replace %run sentences with import using the new databrick public available features (Repo integration and python import):…
Nasgar
  • 859
  • 2
  • 11
  • 26
1
vote
2 answers

Why isn't SQL Endpoints enabled in Azure Databricks?

I want to create an SQL Endpoint as per the documentation: https://learn.microsoft.com/en-us/azure/databricks/sql/admin/sql-endpoints However this option does not appear in the databricks assets drop down: Why not and how can I enable SQL Endpoints…
Brendan Hill
  • 3,406
  • 4
  • 32
  • 61
1
vote
1 answer

how to pass static value into dynamic on basis of column value in azure databricks

how to pass static value into dynamic on basis of column value in Azure Databricks. Currently, I have 13 notebook and its scheduled ,so I want to schedule only one notebook and In addition, data of column( 13 rows) which I defined separate in 13…
siddhesh
  • 41
  • 1
  • 13
1
vote
1 answer

Can someone please redirect me to some study material links for "Databricks certified Professional Data Engineer" certification"?

Can someone please help with the study material for "Databricks certified Professional Data Engineer" certification - the new one that has come out recently ? #databricks #github
azuresnowflake1
  • 135
  • 1
  • 10
1
vote
0 answers

How to create Azure Databricks Job cluster to save some costing compared to Standard cluster?

I have a few pipeline jobs on Azure Databricks that run ETL solutions using standard or high concurency clusters. I've noticed on azure costings page that job cluster is a cheaper option that should do the same thing.…
BI Dude
  • 1,842
  • 5
  • 37
  • 67
1
vote
1 answer

Create Spark context from Python in order to run databricks sql

I've been following this tutorial which lets me connect to Databricks from Python and then run delta table queries. However, I've stumbled upon a problem. When I run it for the FIRST time, I get the following error: Container container-name in…
anthino12
  • 770
  • 1
  • 6
  • 29
1
vote
1 answer

Azure databricks - Do we have postgres connector for spark

Azure databricks - Do we have postgres connector for spark Also, how to upsert/update record in postgres using spark databricks. I am using Spark 3.1.1 When trying to write using mode=overwrite, it truncates the table but recird is not getting…
Arnab Mandal
  • 21
  • 5
1
vote
1 answer

Azure databricks toggle environment vars contain quotes in python

I know there are a lot of questions here about how to handle quotes in environment variables. This question has a different focus so please read on: Before last week we had set our environment variables on our databricks cluster (7.3 LTS, includes…
HeyMan
  • 1,529
  • 18
  • 32
1
vote
2 answers

Get list of all notebooks in my databricks workspace

How do I get a list of all notebooks in my workspace & store their names along with full path in csv file, I have tried using Databricks CLI option but that doesn't seem to have recursive operation. databricks workspace list
Learn2Code
  • 107
  • 1
  • 11
1
vote
1 answer

Connect to Databricks from python via managed identity

I managed to connect to Databricks from python using the following code snippet: from databricks import sql connection = sql.connect( server_hostname='', http_path='', …
anthino12
  • 770
  • 1
  • 6
  • 29
1
vote
2 answers

Reading azure databricks logs json file using azure databricks

I have sent the data bricks logs to storage account by enabling diagnostic setting, Now I have to read those logs using azure data bricks for advance analytics. when I try to mount the path it works but reads wont work . step 1- containerName =…
anuj
  • 124
  • 2
  • 13
1
vote
0 answers

Trouble installing JAR package for Databricks Labs Smolder Project

I attempted to follow the readme here: https://github.com/databrickslabs/smolder When I run the imports in Azure Databricks with runtime 8.4 (includes Apache Spark 3.1.2, Scala 2.12) import…
Pfwangs
  • 41
  • 5
1
vote
1 answer

How to use strsplit on SparkDataFrame

I am taking my first steps in the Azure Databricks world and therefore I have to learn how to use SparkR. [I am coming from data.table] Although I have read a lot of documentation, I think something escapes me on SparkDataFrame. To create a new…
Discus23
  • 471
  • 2
  • 11
1
vote
1 answer

PGP encryption in azure databrick

I need your help badly :D I wrote a code in python with PGP , I have a trusted public key and I could perfectly encrypt my massage with this code, but when I run it on data brick I faced problem : gnupghome should be a directory and it isnt I would…
Marian
  • 11
  • 2
1
vote
2 answers

Read a file as byte string from a Databricks ADLS mount point

The goal is to read a file as a byte string within Databricks from an ADLS mount point. Confirming the ADLS mount point Firstly, using dbutils.fs.mounts() it is confirmed to have the following: ... MountInfo(mountPoint='/mnt/ftd',…
Filip Markoski
  • 333
  • 3
  • 19
1 2 3
…
99 100