Stack Overflow
Stack Overflow

Questions tagged [aws-databricks]

For questions about the usage of Databricks Lakehouse Platform on AWS cloud.

Databricks Lakehouse Platform on AWS

Lakehouse Platform for accelerating innovation across data science, data engineering, business analytics, and data warehousing integrated with your AWS infrastructure.

Reference: https://databricks.com/aws

190 questions
0
votes
1 answer

how to format date in databricks notebook sql

I need to convert date vales that are in mm/dd/yyyy format to yyyy-mm-dd in databricks sql. I tried to_date('eff_date', 'mm/dd/yyyy') then i am getting the correct output format but all the rows month values are defaulting to 01. for eg: 12/01/2016…
user15906795
  • 29
  • 6
0
votes
1 answer

How to exclude Glacier data from input source when reading it via Spark?

I'm trying to exclude Glacier data from the input of my Databricks notebook job (Spark). It basically reads parquet data on S3 through AWS Glue Catalog. I already added excludeStorageClasses to Glue table properties: |Table Properties |…
Finkelson
  • 2,921
  • 4
  • 31
  • 49
0
votes
3 answers

Kafka consumer using AWS_MSK_IAM ClassCastException error

I have MSK running on AWS and I'd like to consume information using AWS_MSK_IAM authentication. My MSK is properly configured and I can consume the information using Kafka CLI with the following command: ../bin/kafka-console-consumer.sh…
Danilo Cairolli
  • 51
  • 4
0
votes
1 answer

auto-suggest stopped working on my notebook

After switching between clusters (detach/attach), the auto-suggest feature stopped working for me on the current notebook I work on and I can't get auto-completion when I type the TAB key after a period, I get: 'no suggestions'. I tried to delete…
Yair Nevet
  • 12,725
  • 14
  • 66
  • 108
0
votes
0 answers

Databricks connect does not work from intellj?

I am trying to use databricks connect to run the spark job on databricks cluster from intellj .I followed the below link documentation. https://docs.databricks.com/dev-tools/databricks-connect.html However I could not make it work with intellj and…
Rajesh Kumar Dash
  • 2,203
  • 6
  • 28
  • 57
0
votes
1 answer

Databricks Serverless Computer - writeback to delta tables

Databricks Serverless Compute - I know this is still in preview and is by request and is only available on AWS. Can this be used for Read and Write (Update) .delta tables [or] is it read-only? And is it good to run small queries (transactional in…
Sreedhar
  • 29,307
  • 34
  • 118
  • 188
0
votes
2 answers

from_json returns null in Apache Spark 3.0

I have a pyspark column of type string containing a array of dictionary. x = {"a":1,"b":[{"type":"abc","unitValue":"4.4"}]} I want to cast the string into array of struct but while doing that the fields in the new column are getting populated…
Saikat
  • 403
  • 1
  • 7
  • 19
0
votes
1 answer

Azure Databricks : The backend could not get session tokens for path /mnt. Did you remove the AWS key for the mount point?

I'm trying to Mount Azure Blob storage containers to DBFS. Implementation is as below. got below following error . Not sure why I'm getting an error Did you remove the AWS key for the mount point? when I'm trying to connect to azure blob…
Sri
  • 35
  • 7
0
votes
1 answer

Error in performing 'show create table' OpenCSV Serde Hive Table in Databricks

I have created a table with OpenCSV Serde in Databricks below DDL :- CREATE TABLE db_name.table_name( col1 String, col2 String) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' WITH SERDEPROPERTIES ( "escapeChar"="\"", …
Vinitkumar
  • 111
  • 5
  • 18
0
votes
2 answers

Set default timezone in Databricks to ESTA

I'm running code in Databricks but somehow unixtimestamp values are not different. On debugging I get to know, because databricks is configured with UTC timezone. Please advise how can I change default timezone to EST in Databricks.
Vinitkumar
  • 111
  • 5
  • 18
0
votes
1 answer

Is there any way to run/execute cells after a certain cell in Databrick notebook?

I'm wondering if it is possible to run cells after a certain cell on Databricks notebook instead of Run All option on top of notebook?
Mario
  • 1,631
  • 2
  • 21
  • 51
0
votes
1 answer

Databricks notebook pass in parameters return result

Hi I am very new to Databricks and wanted some guidance. I trying to pass in some parameters into a Databricks Notebook and I want it to do some simple computations. For example, pass in two parameters, x and y => return x + y as a response. I tried…
Hanad Hassan
  • 11
  • 1
0
votes
1 answer

databricks editor UI issues

When I click on a databricks results display (to copy it), the page refreshes and does a page up. And then it comes back to the same cell. Also when I do ctrl+C on the display cell(copy), the notebook starts execution! What am I doing wrong? Did I…
Blue Clouds
  • 7,295
  • 4
  • 71
  • 112
0
votes
1 answer

Can AWS Glue catalog point to a data location in Azure ADLS?

We are trying configure AWS Databricks Runtime to use the AWS Glue Data Catalog as its metastore. In this environment ,Azure ADLS is one of the source system.In that case,Can AWS Glue catalog point to a data location in Azure ADLS?
Karthikeyan Rasipalay Durairaj
  • 1,920
  • 13
  • 35
0
votes
1 answer

ADLS - Accessing the ADLS from Databricks for SQL mode

In Databricks , we are able to access the ADLS file based on the following authentication code in Python mode .But when I tried to Authenticate for SQL mode getting below error . Please help us to get information on how to declare authentication in…
Karthikeyan Rasipalay Durairaj
  • 1,920
  • 13
  • 35
1 2 3
…
12
13