1

The following code is intended to run unit tests in Databricks notebooks, using pytest.

import pytest
import os
import sys

repo_name = "Databricks-Code-Repo"

# Get the path to this notebook, for example "/Workspace/Repos/{username}/{repo-name}".
notebook_path = dbutils.notebook.entry_point.getDbutils().notebook().getContext().notebookPath().get()

# Get the repo's root directory name.
repo_root = os.path.dirname(os.path.dirname(notebook_path))
    
# Prepare to run pytest from the repo.
os.chdir(f"/Workspace/{repo_root}/{repo_name}")
print(os.getcwd())
    
# Skip writing pyc files on a readonly filesystem.
sys.dont_write_bytecode = True
    
# Run pytest.
retcode = pytest.main([".", "-v", "-p", "no:cacheprovider"])
    
# Fail the cell execution if there are any test failures.
assert retcode == 0, "The pytest invocation failed. See the log for details."

This code snippet is in the guide provided by Databricks. However, it produces the following error:

PermissionError: [Errno 1] Operation not permitted: '/Workspace//Repos/<email_address>/Databricks-Code-Repo/Databricks-Code-Repo'

This notebook is inside Databricks Repos. I have two other notebooks:

  • functions (where I have defined three data transformation functions);
  • test_functions (where I have defined test function for each of the data transformation functions from the previous notebook).

I get that the error has something to do with permissions, but I can't figure out what is causing it. I will appreciate any suggestions.

PieCot
  • 3,564
  • 1
  • 12
  • 20
lyubol
  • 119
  • 9
  • most probably `pytest` is trying to write something to the WSFS representing Databricks Repo, maybe `__pycache__` files. But have you looked onto using Nutter to test notebooks directly? Like in this demo: https://github.com/alexott/databricks-nutter-repos-demo – Alex Ott Dec 27 '22 at 12:08
  • 1
    FWIW, I am facing the same issue and I can assure that I am not trying to write anything. I am merely trying to confirm that the path exists. I am pretty sure this used to work, though, so maybe Databricks changed some backend stuff that broke compatibility with `os.path`/`pathlib` – Bjarne Thorsted Jan 04 '23 at 12:03

1 Answers1

0

I know it is silly, but I restarted the cluster and it worked.

Yunnosch
  • 26,130
  • 9
  • 42
  • 54
Henrique Brisola
  • 326
  • 3
  • 12