0

I am using AWS EMR and I am using an open-source delta lake.

In Python, dataframe.write.format('delta').save() works fine.

But I want to use it in SQL. I tried to create a delta table in SQL as below.

spark.sql('''
CREATE OR REPLACE TABLE test.foo
   (name string)
   USING delta
   LOCATION 's3://<bucket_name>/test/foo'
''');

But when I try to INSERT, an error is raised.

spark.sql('INSERT INTO test.foo (name) VALUES ("bar")');

ERROR: org.apache.spark.sql.AnalysisException: Path does not exist

Tables were created in Glue Metastore, but nothing was created in s3://<bucket_name>/test/foo in S3.

Is there any way to create a table in SQL? :)

dahuin
  • 67
  • 7

1 Answers1

1

To me it looks like you use the wrong name: test.sql_delta. If u use sql and created a sql table it will reference to your physical metastore through the SQL table name you just created.

code should be:

spark.sql('INSERT INTO test.foo (name) VALUES ("bar")')

SQL version:

%SQL
INSERT INTO test.foo (name) VALUES ("bar")
Jurriëndg
  • 11
  • 2