3

I have a DAG where I want to have multiple separate SnowflakeOperator's use the same snowflake SESSION. The reason being, I want them all to use the same TEMPORARY table's.

Unfortunately, even when using the same airflow snowflake connection, usage of session like CLIENT_SESSION_KEEP_ALIVE and heartbeat along with ensuring the default session timeout is set high enough, the operator always creates a brand new session with each operator.

This forces me to either pass all of my queries in a array to one operator (not always possible if i want to do something in the middle) OR to use TRANSIENT tables for which I need to make sure get cleaned up and that multiple instances of the run don't tread all over eachother.

Has anyone managed to do this?

Callum Lamb
  • 31
  • 1
  • 2

0 Answers0