0

I use Azure databricks and databricks CLI to manage it. When I try to copy some local folders and files to databricks DBFS, at first folder it copied with folder and files ( as same as local ). But for the second folders, it just copied files in it not the folder, so it looks something below.

Imagine my folder named 'animal' contains two subfolders like below and I tried to copy animal folder with recursive true.

animal folder contains below subfolders and files in it:

  • animal\land\gorilla.txt
  • animal\land\tiger.txt
  • animal\land\lion.txt

animal folder contains below subfolders and files in it:

  • animal\sea\fish.txt
  • animal\sea\shark.txt
  • animal\sea\whale.txt

Above are saved as below in databricks dbfs:

  • animal\land\gorilla.txt
  • animal\land\tiger.txt
  • animal\land\lion.txt
  • animal\fish.txt
  • animal\shark.txt
  • animal\whale.txt

Below is my command,

databricks fs cp -r 'C:myname/work/animal/' dbfs:/earth/organism

I expect something below in databricks dbfs,

  • /earth/organism/land/gorilla.txt (and all landfiles)
  • /earth/organism/sea/whale.txt (and all sea files but sea folder not even exist )

What Iam doing wrong, Can someone correct me?

Akash
  • 29
  • 8
  • Hard to help with available info. Post output as you see from CLI. Like run `dir /S animal`, `databricks fs ls dbfs...`, `databricks cp -r animal dbfs...`, `databricks fs ls dbfs...` and so on. Then post everything you see. – Kashyap Jun 07 '23 at 20:12

1 Answers1

1

As per my uderstanding In your command

databricks fs cp -r **'C:myname/work/animal/'** dbfs:/earth/organism

Ignore the local folder path in String and try copying the folder and sub folder into DBFS.

I have tried same.

C:\Users\xxxxx\Desktop\Animal>databricks fs cp -r C:/Users/xxxxxx/Desktop/Animal dbfs:/FileStore/tables/Animal/

enter image description here At Databricks DBFS: enter image description here

enter image description hereenter image description here

DileeprajnarayanThumula
  • 643
  • 2
  • 5
  • Dileep and Kashyap, You both very are correct. It's my mistake not to look root cause. Actually, I dowload ADO build artifacts using "DownloadPipelineArtifact@2" this causes the root cause that It downloads as above issue one with folder another not with folder. But, When I try to run with "DownloadBuildArtifact@1" it works as expected. Importantly, Above Databricks command works as expected like copied all files and folders without any misbehave. Now, I'm gonna charge to "DownloadPipelineArtifact@2" team with issue. Thank you very much Dileep and Kashyap. : ) – Akash Jun 08 '23 at 16:54