Lambda to trigger a DLT pipeline

Question

I have a source file that I load into S3 as a delta file. I can attach a lambda trigger to the file. Is there a way to trigger a DLT pipeline based on the lambda trigger

Answer 1

Yes, you can use DLT REST APIs to start existing DLT pipeline (it should be pre-created, although you can create it via APIs as well if it doesn't exist yet). Another approach is to create a Databricks job around DLT pipeline, and then use Jobs REST API to trigger that job.