I am trying to import data using cloudfiles in databricks. 2 different ways give me errors. With
CREATE OR REFRESH STREAMING LIVE TABLE orders_raw
COMMENT "The raw books orders, ingested from orders-raw"
AS SELECT * FROM cloud_files("${datasets_path}/orders-raw", "parquet",
map("schema", "order_id STRING, order_timestamp LONG customer_id STRING, quantity LONG")
)
I get:
[PARSE_SYNTAX_ERROR] Syntax error at or near 'STRING'.(line 1, pos 9)
== SQL ==
order_id STRING, order_timestamp LONG customer_id STRING, quantity LONG
---------^^^
And with :
CREATE OR REFRESH STREAMING LIVE TABLE orders_raw
COMMENT "The raw books orders, ingested from orders-raw"
AS SELECT * FROM cloud_files("${datasets_path}/orders-json-raw", "json",
map("cloudFiles.inferColumnTypes", "true")
)
The error is:
Caused by: java.lang.IllegalArgumentException: cloudFiles.schemaLocation Could not find required option: schemaLocation. Please provide a schema location using
cloudFiles.schemaLocation
for storing inferred schema and supporting schema evolution.
any help? thanks