Stack Overflow
Stack Overflow

Questions tagged [databricks-repos]

For questions around Databricks Repos for Git integration.

This tag is for questions around Databricks Repos - development best practices, code organization, integration with CI/CD, deployment, etc.

58 questions
2
votes
2 answers

Running a databricks notebook connected to git via ADF independent from git username

In our company for orchestrating of running Databricks notebooks, experimentally we learned to connect our notebooks (affiliated to a git repository) to ADF pipelines, however, there is an issue. As you can see in the photo attached to this question…
Ali Saberi
  • 864
  • 1
  • 10
  • 33
2
votes
1 answer

Working in two different git branches at the same time in Databricks

Is there a possibility to work in two different branches at once on Databricks? If I open one branch in tab window (1) and then open another branch in another tab(2), the first tab window(1) automatically changes the git branch to the other tab(2).…
korilium
  • 33
  • 6
2
votes
2 answers

Iterate over files in databricks Repos

I would like to iterate over some files in a folder that has its path in databricks Repos. How would one do this? I don't seem to be able to access the files in Repos I have added a picture that shows what folders i would like to access (the dbrks &…
Lasse
  • 17
  • 9
2
votes
1 answer

Execute git pull on databricks notebook using CLI and/or API

Using Databricks Repos, you can add a git repo to Databricks and execute git actions such as git pull. This is done by clicking on the branch name in the top left, and clicking the button saying "Pull". I would like to do this without clicking on…
Haterind
  • 1,095
  • 1
  • 8
  • 16
2
votes
2 answers

Call Databricks notebook in a specific branch from Data Factory?

I'm using the new Databricks Repos functionality and in Data Factory UI for the notebook activity you can browse the Databricks workspace and select Repos > username > project > folder > notebook. Is it possible to call a Databricks notebook in a…
baatchen
  • 469
  • 1
  • 7
  • 16
2
votes
1 answer

Databricks Import/Copy Data from python lib inside repo

i am facing a little challenge while trying to implement a solution using the new repo functionality of databricks. I am working in a interdisziplinairy project which needs to be able to use python und pyspark code. The python team already builded…
AirCoder
  • 21
  • 2
1
vote
1 answer

Databricks x Azure Devops: Can I create a devops pipeline to update databricks repo post merge?

I currently have a databricks and azure devops integration. At the current time, we develop on the databricks workspace under the 'dev' repo/folder connected to the azure 'dev' repo. We also have a 'Prod' repo/folder and 'main' azure repo. When the…
Sam Richardson
  • 15
  • 3
1
vote
0 answers

Github Integration with Azure Databricks

I followed the steps detailed in the official documentation (https://docs.databricks.com/repos/repos-setup.html). However when I try to make a repo in Databricks - or checkout code from the Repo via a Databricks workflow I get the following…
James Cooke
  • 1,221
  • 1
  • 14
  • 27
1
vote
1 answer

Files in Repos enabled but not working / import modules using databricks feature Repos

I follow all the steps to organize .py files and import them as module using this new DBR feature Files in Repos, but Im still getting same error "No module named". enter image description here What I've done: check cluster version . Its a 9.1. I…
jccorrea
  • 11
  • 2
1
vote
2 answers

Databricks repos notebook deleted by mistake (not commited, not versioned), is posible to restore it?

Accidentally i deleted a notebook from my databricks repo before any commit, I expected to restore it from trash folder but the notebook is not there. Is posible to restore from any directory on the server or something like that?
Santiago
  • 11
  • 1
1
vote
0 answers

How to connect to Azure Devops git repo through databricks?

I want to create a python notebook in databricks that will do the following - Connect to Azure Devops Git repo Make couple of changes in a yaml file Commit the changes in the master branch Push the changes back to repo I tried the below code to…
user19930511
  • 299
  • 2
  • 15
1
vote
1 answer

Unit testing in Databricks notebooks

The following code is intended to run unit tests in Databricks notebooks, using pytest. import pytest import os import sys repo_name = "Databricks-Code-Repo" # Get the path to this notebook, for example…
lyubol
  • 119
  • 9
1
vote
0 answers

How to read a .sql file (not notebook) in a DataBricks repo to a String in Scala?

I can't seem to accomplish this is scala, seems easy in python, bash and R, but im trying to keep our code base in scala. I know the repo is set up correctly because when I execute %sh cat /absolute/path/to/file/in/repo I get the file contents…
hello_friend
  • 5,682
  • 1
  • 11
  • 15
1
vote
0 answers

Can libraries be included in databricks repos?

I tried to move a cluster library into a databricks repo and I got this error message: It seems to be not supported, but then I found this question in the documentation: I was able to add the libraries moving the whole folder where I put them, but…
Playing With BI
  • 411
  • 1
  • 9
  • 20
1
vote
1 answer

Notebook in Azure Databricks Repos cannot show difference

I am trying to commit and push my change to the branch, I cannot load the difference. I haven't changed many cells and each cells doesn't exceed the 500 lines in the notebook file. I am wondering why this happens and how to solve it?
Youshikyou
  • 365
  • 1
  • 8
1
2
3 4