Stack Overflow
Stack Overflow

Questions tagged [databricks-sql]

Questions about Databricks SQL

For questions about Databricks SQL - a serverless data warehouse on the Databricks Lakehouse Platform that lets you run all your SQL and BI applications at scale with improved performance, a unified governance model, open formats and APIs, and your tools of choice

357 questions
2
votes
1 answer

OBJECT_DEFINITION for VIEW in Azure Databricks

As the Title says, is it able to query object definition of the view in Databricks using SQL, especially for Azure Databricks? like when using SQL Server, I'm able to query the view definition using function OBJECT_DEFINITION or Stored Procedure…
tmo
  • 117
  • 2
  • 8
2
votes
2 answers

Automating Permissions for Databricks SQL Tables or Views

Trying to automate the setup of Databricks SQL. I have done it from the UI and it works, so this is a natural next step. The one thing I am unsure about is how to automate granting of the access to SQL tables and/or views using REST. I am trying to…
Saugat Mukherjee
  • 778
  • 8
  • 32
2
votes
1 answer

Coalesce() function on databricks

I want to replace null values from a column to 0 using coalesce() function instead of isNULL(). Can anyone tell me how ?
Abhijit Wadkar
  • 21
  • 2
2
votes
1 answer

Unable to cast string to date when inserting on Databricks SQL

I have a table created like so : CREATE TABLE IF NOT EXISTS MyDataBase.Table ( `date` DATE, `name` STRING, `isTarget` BOOLEAN ) USING DELTA LOCATION '/mnt/path/to/folder/' I need to manually insert values in the table using SQL. I…
Karzyfox
  • 319
  • 1
  • 2
  • 15
2
votes
1 answer

How to create a new table with the results of SHOW TABLES in Databricks SQL?

I want to do aggregations on the result of SHOW TABLES FROM databasename Or create a new table with the result like CREATE TABLE database.new_table AS ( SHOW TABLES FROM database ); But I'm getting multiple different errors if I try to do…
robin_xxx
  • 21
  • 2
2
votes
1 answer

Databricks/Spark SQL - how to fetch data in a map type field

In a CDP datalake the profiles table features the identityMap field: [IdentityType -> IdentityValue] A sample value looks like the following: ["crmID" -> "{(NULL,crm123,NULL)}", "loyaltyID" -> "{(NULL,loy456,NULL)}", "accountID" ->…
Alberto De Caro
  • 5,147
  • 9
  • 47
  • 73
2
votes
2 answers

AnalysisException: Operation not allowed: `CREATE TABLE LIKE` is not supported for Delta tables;

create table if not exists map_table like position_map_view; While using this it is giving me operation not allowed error
Saswat Ray
  • 141
  • 3
  • 14
2
votes
1 answer

How to manage Databricks SQL Endpoints from application / command line?

Is there a way to start, stop, list available SQL Endpoints in Databricks SQL? With "Can Use" permission it's possible to start a SQL Endpoint while connecting to an endpoint using the Databricks JDBC driver, but have no idea how to stop it (not to…
Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
2
votes
1 answer

COPY INTO: How to add a partitioning?

The command COPY INTO from Databricks provides an idempotent file ingestion into a delta table, see here. From the docs, an example command looks like this: COPY INTO delta.`target_path` FROM (SELECT key, index, textData, 'constant_value' FROM…
tschmelz
  • 480
  • 1
  • 4
  • 10
2
votes
2 answers

DATABRICKS SQL - can't read data from partitioned parquet file

I'm trying to read parquet files structured as: filename/year=2020/month=12/day=1 files are under the following Mounted AzureStorage as following logic: /mnt/silver/root_folder/folder_A/parquet/year=2020/month=01/day=1 I'm trying to create a table,…
Vinny
  • 49
  • 7
2
votes
1 answer

Reports/Visualization on top of Databricks SQL Analytics(Redash) query history?

Is it possible to directly query the query history (logs) from Databricks SQL Analytics workspace and develop some reports on query metrics?
Venkat Tumu
  • 21
  • 2
1
vote
0 answers

How to expand a single table row to multiple rows using select rows in another table matching a timestamp condition across both tables

Introduction (DB-fiddle here: https://dbfiddle.uk/o5gsnxpt) Say you manage busses and that you have two tables, bus_cancellations and bus_timetables: The table bus_cancellations contains all cancellations of bus journeys (only one shown here) where…
QueryingQuail
  • 45
  • 6
1
vote
2 answers

How do I pull table name with column name list using databricks sql?

I am using Azure Databricks INFORMATION_Schema. Need list all tables and columns using SQL. I used SHOW TABLES command for viewing all tables within a database. SHOW COLUMNS command for viewing all columns within a table. What am I missing? How do…
Bricks
  • 11
  • 1
1
vote
1 answer

Databricks enforce primary and foreign key constraints in Unity Catalog

Does anyone have an idea about databricks delta tables in unity catalog supporting primary and foreign key constraints by enforcing them and not just declaring for informational purposes? I'm trying to create delta tables in unity with foreign key…
dsavdb
  • 13
  • 2
1
vote
1 answer

How to capture the number of rows inserted/updated to Delta table in Databricks

I am trying to capture the number of rows inserted/updated for a table in databricks Please see the below screenshot for more details. I need to pull the number 31 from num_affected_rows
bigdata techie
  • 147
  • 1
  • 11
1 2
3
…
23 24