Stack Overflow
Stack Overflow

Questions tagged [dbutils]

Databricks Utilities (dbutils) works with object storage to chain and parameterize notebooks and work with secrets.

Documentation: Databricks Utilities | Databricks on AWS

44 questions
0
votes
0 answers

How to use scala variables in one cell in %sh command in azure data bricks?

I was running some shell script in azure databricks using %sh magic command. I have a requirement to pass few arguments to that shell script. I was using Scala as my primary language. I have declared few variables in Scala and passing those…
venkat
  • 111
  • 1
  • 1
  • 11
0
votes
0 answers

Return ephemeral job exception in databricks when using dbutils.notebook.run() function

I am trying to return the exception message from an ephemeral notebook that is run with the dbutils.notebook.run(file_path) function in databricks. My notebook looks like this I'm wondering if I can get the exception message to output in the current…
nleffell
  • 1
  • 1
0
votes
1 answer

Databricks update files which name is on DataFrame

I have a DataFrame like: input_df = self.spark.createDataFrame( data=[ ("01", "file_name_1"), ("02", "file_name_2"), ("05", "file_name_5"), ], schema=( …
jalazbe
  • 1,801
  • 3
  • 19
  • 40
0
votes
2 answers

Easily entering arguments from dbutils.notebook.run when using a notebook directly

I'm calling a notebook like this: dbutils.notebook.run(path, timeout, arguments) where arguments is a dictionary containing many fields for the notebook's widgets. I want to debug called notebook interactively: copy/pasting the widget parameters…
Michael Grazebrook
  • 41
  • 5
0
votes
0 answers

why pool.connection().cursor().execute() is a warning operation?

I'm new to python and learning DB operation with DBUtils. Why pool.connection().cursor().execute() would release the connection too early for reuse? If you don't need it any more, you should immediately return it to the pool with db.close(). You…
Holinc
  • 641
  • 6
  • 17
0
votes
0 answers

Multiprocess executors in Python

I am trying to run a report on my storage of Azure Gen2 Data lake. I have written a below recursive function that goes inside every folder and list files till last level. def recursive_ls(path: str): """List all files from path…
Harshit
  • 560
  • 1
  • 5
  • 15
0
votes
0 answers

Marge Move/Copy Large data file in dbutils using a Notebook in pyspark

Writing the dataframe as a tsv in databricks file system (DBFS) with huge data size (30GB to 1TB). I am currently using the below code df.coalesce(1).write.format("csv").option("delimiter", "\t").option("nullValue",None).option("header",…
ram
  • 323
  • 4
  • 12
0
votes
1 answer

Installing Maven library on Databricks via Python commands and dbutils

On Databricks I would like to install a Maven library through commands in a Python Notebook if its not already installed. If it were a Python PyPI library I would do something like the following: # Get a list of all available library…
sbs0202
  • 139
  • 3
  • 8
0
votes
1 answer

How to kill parallel execution of Databricks notebooks?

I am currently using Python's Threading to parallelize the execution of multiple Databricks notebooks. These are long-running notebooks, and I need to add some logic for killing the threads in the case where I want to restart the execution with new…
sondrelv
  • 111
  • 2
  • 6
0
votes
1 answer

DBFS : How to get Last access time of DBFS file in Datarbricks

Is it possible to get the time when file was opened last time as like ls -ltu list all the files, showing and sorting by access time in Unix by using of dbutils. This we required to get the stat about ADLS file in Databricks note.
Karthikeyan Rasipalay Durairaj
  • 1,920
  • 13
  • 35
0
votes
2 answers

Can i return more than 1 value from databricks notebook in a single command?

I have a set of values to return as output from my databricks notebook. Can anyone sugggest a way to do that in an efficient and easy way?
Samyak Jain
  • 155
  • 1
  • 2
  • 8
0
votes
2 answers

How to generate a databricks privileged token which is valid more than 48 hours

Would like to run a local Jupiter notebook connecting to Azure databricks cluster, and need to use dbutils to get secrets. This requires to save a privileged token locally and it is only lasting for 2 days. Is there any way to generate token longer…
zzzk
  • 135
  • 10
-1
votes
1 answer

Snowflake Jdbc rs.beforeFirst() method error

I am trying to get test by using snowflake jdbc .I connected . but when I try to get some data from snowflake I got this error.net.snowflake.client.jdbc.SnowflakeLoggedFeatureNotSupportedException. I asked the snowflake support they said you are…
Aykut.Koc
  • 1
  • 2
-2
votes
1 answer

net.proteanit.sql for gradle

I'm trying to open same project in 3 build tools: Maven, Ant and Gradle. It's simple project with sql database, I've already done this via Ant and Maven, unfortunately after adding rs2xml.jar to project module(as I added in Ant and Maven), and…
hajscube
  • 1
  • 3
1 2
3