How do I dynamically get the database name from a DLT pipeline?
Asked
Active
Viewed 216 times
1
-
I'm not sure that there is an easy way to do that. For what purpose do you need it? – Alex Ott Dec 21 '22 at 18:12
-
separation of dev vs production databases – jencake Dec 23 '22 at 21:44
-
let me elaborate, I have to separate a single DLT pipeline into two separate DLT pipelines because DLT doesn't have support for ignoreChanges. ignoreChanges is needed because we have to delete source data in order to comply with data privacy laws. the second DLT pipeline would need to load the output of the first DLT pipeline with spark.readStream(f"{database}.{table_name}") and I'd like to easily switch to a dev database without impacting the production pipeline. – jencake Dec 23 '22 at 22:00