mkdir -p skips creating a middle directory on the Azure Datalake

Asked Mar 22 '19 at 20:22

Active Mar 22 '19 at 20:47

Viewed 185 times

Why does this consistently make a path in Databricks

%sh
/dbfs/mnt/datalake/data/staging/steve/3/14

No matter what it drops creating the 2019.

If I leave out /2019 then it drops creating the 3.

Here is my Cmd

mkdir -p /dbfs/mnt/datalake/data/staging/steve/2019/3/14

edited Mar 22 '19 at 20:47

David Makogon

asked Mar 22 '19 at 20:22

Steve Lyle-CSG

I don't know what Azure Datalake is, but if `mkdir` on it is supposed to be POSIX compliant then you need to report it as a bug [POSIX `mkdir`](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/mkdir.html) – David C. Rankin Mar 23 '19 at 02:45
1

@SteveLyle-CSG - I think you replied to the wrong person. – David Makogon Mar 25 '19 at 01:47
@DavidC.Rankin Thanks Dave. The %sh command is being executed in a Databricks notebook, which runs on Linux Ubuntu, and is an action against Azure Data Lake Storage (ADLS) integrated to Ubuntu as a mounted storage. This is Microsoft's principal architecture for working with Spark/Hiv in Azure. – Steve Lyle-CSG Mar 25 '19 at 16:01

0 Answers0