0

I want to check the possibility of triggering dbt models after snowpipe load data into Snowflake database. I have a requirement where realtime xml data comes everytime and I'll use snowpipe to ingest that data into snowflake. I want to know if there is a way to run dbt models that will load the variant field data to raw tables and run subsequent dbt models as well. Kindly suggest the alternatives as well. I dont have any orchestration tools used to do this.

Thanks,

V.

Ven Raj
  • 47
  • 6
  • I think using an orchestrator is the best way to go here. If you are open to using orchestration tools to do this, I'd recommend Dagster. – Kay Aug 17 '23 at 14:33
  • Is there any other way to do it from snowflake or python? – Ven Raj Aug 17 '23 at 23:30
  • You can process the incoming data to a materialized table on schedule (by using dbt cloud or any other orchestrators as Kay suggested). For satisfying the realtime need, you can build a view which unions unprocessed (new data loaded from Snowpipe) and the materialized table. But do not forget to process the new data on a schedule otherwise your views will perform inefficiently in the long run – ozgenbaris Sep 01 '23 at 08:16

0 Answers0