Stack Overflow
Stack Overflow

Questions tagged [databricks-sql]

Questions about Databricks SQL

For questions about Databricks SQL - a serverless data warehouse on the Databricks Lakehouse Platform that lets you run all your SQL and BI applications at scale with improved performance, a unified governance model, open formats and APIs, and your tools of choice

357 questions
2
votes
2 answers

How do column data types affect join performance in SPARK or Databricks environment?

I was recently introduced to DBT tool. One downside of the tool is that you cannot create an identity column (surrogate keys) as sequences. You can only generate hash of columns to uniquely identify rows. Due to that reason, I was trying to find out…
Jeyhun
  • 83
  • 1
  • 5
2
votes
0 answers

How to see results of multiple SQL queries in Databricks?

I have a cell with multiple SQL queries like MERGE, SELECT etc. When I run the notebook through a workflow TASK I can see the output of the last run SQL query for a particular cell only. How to see output for each queries?
Anirban Nag 'tintinmj'
  • 5,572
  • 6
  • 39
  • 59
2
votes
0 answers

I want to use parameterized query in @databricks/sql node.js

Query = "select * from repos where id="sj" parameterized = "select * from repos where id = ?" result - executestatement(parameterized,"sj") I want to pass parameters to the sql query for @databricks/sql module but I am unable to figure out a way.…
Himanshu Sethi
  • 21
  • 1
2
votes
0 answers

Spark - nondeterministic expressions not allowed using Delete FROM and exists

The goal of this query is to delete the records that have the same keys of the new dataframe. This is the query that I'm executing: val op=spark.sql(s""" DELETE FROM TABLE1 AS t WHERE EXISTS ( SELECT 1 FROM TABLE2 AS s WHERE…
ultraInstinct
  • 4,063
  • 10
  • 36
  • 53
2
votes
1 answer

Failure when creating a table during job run on Databricks

I have been doing a left join on two tables in SQL (table A contains billions of rows and table B contains millions of rows) and creating a new table (table C) from the result of the join. I am using a XXL SQL Warehouse on Databricks with two…
Alessio
  • 21
  • 3
2
votes
1 answer

Difference Between df.wirte and CREATE TABLE USING

I have always been under the impression that the following code create a Delta table, data.write.format("delta").save("/path/to/delta-table") This creates the files, sure, however, I noticed today that when I look at the Data section of Databricks,…
Minura Punchihewa
  • 1,498
  • 1
  • 12
  • 35
2
votes
1 answer

Databricks SQL database creation with location Azure Data Lake

How to create a database in Databricks hive metastore with location adls and abfss (without mounting) ? Something like this create database if not exists adb_raw location "abfss://container@stgaccount.dfs.core.windows.net/01_RAW/flat_files/" will…
Pirvu Georgian
  • 657
  • 1
  • 12
  • 37
2
votes
1 answer

How to group by continuous time periods and sum a column in sql

id| startdate | enddate | team | score 1| 2011-09-25 | 2012-03-31 | blue | 380 1| 2012-04-01 | 2012-07-31 | blue | 147 1| 2012-08-01 | 2013-01-31 | green| 217 1| 2013-02-01 | 2015-02-28 | green| 1540 1| 2015-03-01 | 2015-09-30 | green| 300 1|…
Martin
  • 55
  • 6
2
votes
1 answer

jest.mock() not working, tests are still pulling code from node modules

I am trying to mock the @databricks/sql package using the jest.mock function and it continually pulls from the node modules package instead of the code I pass into the function. My test file: import mockResponse from "./mockResponse.json"; const { …
Alisha Edington
  • 23
  • 3
2
votes
0 answers

How to show tables in databricks without views?

The SHOW TABLES IN mydb query lists tables and views, while SHOW VIEWS IN mydb only lists views. Is there any way to list only the tables of a given database ?
BenjaminC
  • 27
  • 4
2
votes
1 answer

how to resolve Arithmetic overflow error in Databricks SQL Editor

In Databricks SQL editor , I am trying to get the SUM of few 'Bigint' and 'Double' data type columns of a table. But am getting below error. Job aborted due to stage failure: Task 0 in stage 29.0 failed 4 times, most recent failure: Lost task 0.3…
kiruba
  • 129
  • 5
2
votes
1 answer

Table Comment missing on Azure Delta Table

I have created a delta table in azure databricks with COMMENT as below. DROP TABLE IF EXISTS dbtest.test_antony; CREATE TABLE IF NOT EXISTS dbtest.test_antony( db_name STRING COMMENT 'EN: Database Name ', table_name STRING COMMENT 'EN: Table…
Antony
  • 970
  • 3
  • 20
  • 46
2
votes
1 answer

Different Methods for Creating EXTERNAL TABLES Using Spark SQL in Databricks

I believe I understand the basic difference between Managed and External tables in Spark SQL. Just for clarity, given below is how I would explain it. A managed table is a Spark SQL table for which Spark manages both the data and the metadata. In…
Minura Punchihewa
  • 1,498
  • 1
  • 12
  • 35
2
votes
1 answer

Delta table : COPY INTO only specific partitioned folders from S3 bucket

I have one landing S3 buckets, which will receive each day some data, partitioned by date : s3:/my_bucket/date=2020-01-01/my_data.txt s3:/my_bucket/date=2020-01-02/my_data.txt s3:/my_bucket/date=2020-01-03/my_data.txt I did create a first test…
Gohmz
  • 1,256
  • 16
  • 31
2
votes
1 answer

Which data type is equal to bit in the databricks sql

Column_name bit in mssql When creating table in the databricks bit data type is not supported Which data type can I use
Krishna Teja
  • 21
  • 1
1
2
3
…
23 24