Stack Overflow
Stack Overflow

Questions tagged [azure-databricks]

For questions about the usage of Databricks Lakehouse Platform on Microsoft Azure

Overview

Azure Databricks is the Azure-based implementation of Databricks, which is a high-level platform for working with Apache Spark and includes Jupyter-style notebooks.

Azure Databricks is a first class Azure service and natively integrates with other Azure services such as Active Directory, Blob Storage, Cosmos DB, Data Lake Store, Event Hubs, HDInsight, Key Vault, Synapse Analytics, etc.

Related Tags

4095 questions
1
vote
0 answers

You may get a different result due to the upgrading of Spark 3.0: Fail to parse 'JUN 2001' in the new parser

I'm working on converting a string to date in SparkSQL. The string format is "Jun 2001', I tried to convert to date format as 'MMMyyyy' by using the query below SELECT TO_DATE(CAST('JUN 2001'AS STRING),'MMMyyyy') However, it kept giving me…
Rosa Xue
  • 11
  • 2
1
vote
1 answer

Running another notebook inside a notebook which is connected to git

In our company we learned to connect our notebook to git repository, however there is an issue. Running a notebook inside another notebook. This is the way that we have discovered, however this solution cant go into production because it included an…
Ali Saberi
  • 864
  • 1
  • 10
  • 33
1
vote
1 answer

How to use PipelineParameter in DatabricksStep (Python)

I've created an AML Pipeline with a single DatabricksStep. I've need to pass a parameter to the Databricks notebook when I run the published pipeline. When I run the published pipeline, the Databricks steps always take the default value of the…
Giorgio
  • 31
  • 4
1
vote
1 answer

Databricks - transfer data from one databricks workspace to another

How can I transform my data in databricks workspace 1 (DBW1) and then push it (send/save the table) to another databricks workspace (DBW2)? On the DBW1 I installed this JDBC driver. Then I tried: (df.write .format("jdbc") .options( …
romanzdk
  • 930
  • 11
  • 30
1
vote
1 answer

Import ML Model from ADLS to Azure ML using Databricks

I have stored one ml model in my ADLS and I want to register the model to Azure ML using databricks. Tried to use the following codes to register my ml model but keep encountering an error that the path cannot be found. I have mount the storage to…
whaleshark
  • 11
  • 2
1
vote
1 answer

java.util.concurrent.TimeoutException error in spark Databricks

Im using the below custom receiver to consume data from Rabbitmq in Spark-Scala. Below is my code. def onStart() { // Start the thread that receives data over a connection new Thread("Socket Receiver") { override def run() { receive()…
SanjanaSanju
  • 261
  • 2
  • 18
1
vote
0 answers

403 error when connecting to S3 using Scala in Azure databricks, Python with boto3 works fine

I have been getting intermittent issues with when trying to read from an S3 bucket from Databricks in Azure. It can sometimes go months with out working, suddenly work temporarily, and stop again. The Scala code is as follows: val access_key =…
ewong18
  • 144
  • 1
  • 2
  • 10
1
vote
0 answers

Where can I check the running cost of a Databricks SQL Endpoint?

Where can I check the running cost of a Databricks SQL Endpoint? I checked the Cost Analysis and there is an item called SQL database, is that it?
Gabriel
  • 5,453
  • 14
  • 63
  • 92
1
vote
2 answers

How to store SparkR result into an R object?

Still new to the world of Azure Databricks, the use of SparkR remains very obscure to me, even for very simple tasks... It took me a very long time to find how to count distinct values, and I'm not sure it's the right way to go…
Discus23
  • 471
  • 2
  • 11
1
vote
1 answer

data.table fread in Databricks Spark

I've done a fair amount of searching and haven't come across any solid info regarding the use of the data.table package in the Databricks environment. Myself and other colleagues have carried out tests in Databricks trying to use the data.table…
Foxhound013
  • 301
  • 3
  • 13
1
vote
1 answer

Create a Database with name from variable on Databricks (in SQL, not in Spark)

How to create a database with a name from a variable (in SQL, not in Spark) ? I've written this : %sql SET myVar = CONCAT(getArgument('env'), 'BackOffice'); CREATE DATABASE IF NOT EXISTS myVar ("env" is a dropdown widgets) But it creates me a…
Salah K.
  • 133
  • 12
1
vote
2 answers

databricks init script UI is empty

In the databricks cluster configuration UI, I am trying to add an init script. I have stored the script in the DBFS, however the init script UI has no drop down or obvious way to select this file. What am I missing? I have followed the instructions…
PFE
  • 39
  • 2
1
vote
1 answer

How to save and reuse my model in databricks

I have complete pipeline in place from data inputs to final predictions. I have 10 models in place (wrote function to automate the whole process) then , Choose 1 model based on accuracy. Now, I need to store all the model(because any model can have…
Amit Kumar
  • 154
  • 12
1
vote
1 answer

Deploy repository to new databricks workspace

I am developing my code in a databricks workspace. Using integration with Repos, I use Azure DevOps to version control my code. I would like to use Azure Pipelines to deploy my code to a new test/production environment. To copy the files to the new…
Larsq
  • 315
  • 2
  • 12
1
vote
1 answer

Geomesa on Databricks - library installation fails

I would like to install the Geomesa library on Databricks, but choosing any geomesa library goes automatically to "failed". I have tried all of the available versions. What am I doing wrong?
Olgaraa
  • 65
  • 1
  • 9
1 2 3
…
99 100