Hey so I am using Airflow 2.5.3 and Dynamic TaskGroup Mapping so I can iterate over rows in a table and use the values in those rows as parameters in this group of tasks.
That is all working fine, and I am getting close to completing what I need to accomplish here, but now at a bit of a snag.
I initially have a few tasks that will grab the list of "active campaigns" so it knows what to iterate over.
Up to this point I had this data hardcoded. So now that I am pulling it from an actual datasource in my tasks, it does not seem I can actually apply the results to my dynamic taskgroup?
dt_group = data_transformation_tasks.expand_kwargs(format_active_campaigns_dict())
When I set it like this, it just runs that "format_active_campaigns_dict" task IMMEDIATELY and of course since it is dependent on previous tasks, it returns NONE, and then once the initial tasks complete and begins the dynamic taskgroup, of course it fails as it already claims there was NONE passed as arguments.
How can I get this TaskGroup to only run the function and get its return value once its reached in its dependencies?
run_active_campaigns_query >> get_active_campaign_data >> format_active_campaigns_dict() >> dt_group
run_active_campaigns_query >> get_active_campaign_data >> dt_group
Both those return None immediately.
I have attempted to skip the task initially until the taskgroup is made, but couldn't get that to work using trigger_rules?
I tried to change order of dependencies.
I did not know if there was a way to use xcoms or similar when setting the dynamic taskgroups arguments...
