0

I am trying to create symlink on databricks delta table which is on ADLS(Azure) with the below command.

%sql
%sql
GENERATE symlink_format_manifest FOR TABLE schema_name.`dbfs:/filepath`;

which fails with the below error: Error in SQL statement: NoSuchTableException: Table or view 'dbfs:/filepath' not found in database 'schema_name';

SunithaM
  • 61
  • 1
  • 7

1 Answers1

1

You should be referring to a delta table either by using

delta.`abfss://<file_system>@<account_name>.dfs.core.windows.net/<path>/<file_name>`

or when your ADLS is mounted to you DBFS:

delta.`/mnt/<path>/<file_name>`

or when using a hive table

delta.`/mnt/<path>/<file_name>`

you can't however link to a file on the filesystem that way

Dr. Casual
  • 418
  • 3
  • 10
  • Thanks Dr.Casual, Yes ADLS is mounted DBFS. When you say file_name am i going to pass all file names? because i have more than one parquet file in ADLS within the table name folder(Eg: mnt/delta_talbe//parque1, parque2.... – SunithaM Mar 22 '21 at 22:13
  • You could pass the folder where the files are in, spark is then smart enough to loop over the files :) – Dr. Casual Mar 23 '21 at 07:21
  • tried that as well looks like it doesn't support Azure storage(it throws error saying it support only AWS S3). I was asked to set the property but not sure how we should do SET spark.databricks.delta.symlinkFormatManifest.fileSystemCheck.enabled = false. I also tried python mode from delta.tables import * deltaTable = DeltaTable.forPath(path = 'dbfs:/mnt/path/', sparkSession = spark) deltaTable.generate('symlink_format_manifest') which thrown warning on deltaTable.generate('symlink_format_manifest'). – SunithaM Mar 23 '21 at 08:28
  • Just set it as the error suggests as so `spark.conf.set("spark.databricks.delta.symlinkFormatManifest.fileSystemCheck.enabled", 'false')`. – Triamus Feb 09 '22 at 05:58