How to Configure Azure Data Factory with multiple schedules

Question

Can someone let me know if its possible configure ADF to be triggered from multiple schedules. For example I would like to execute a pipeline on:

Last Month End e.g 31/05/2023
Last Quarter End e.g. 31/03/2023
Last Year End e.g. 31/12/2021
Last Month End but one e.g 30/04/2023
Last Quarter End but one e.g.30/09/2022

I have looked at scheduling, but I'm not sure if ADF scheduling can trigger on multiple schedules as I have described.

If it's not possible to include all of the above schedules in a single trigger, can someone let me know how I would configure a single trigger for Last Month End but one e.g 30/04/2023

I don't think a single trigger can do what you want. I use Logic Apps to manage complex trigger requirements, check out this question and answer for an example: https://stackoverflow.com/questions/62134686/can-i-trigger-my-azure-data-factory-pipeline-in-5-working-day-between-9-am-to-6/ — Joel Cochran, Jun 05 '23 at 13:52
Hi @JoelCochran, thanks for getting in touch. Could it not even provide the following single trigger ```Last Month End but one e.g 30/04/2023``` — Patterson, Jun 05 '23 at 13:55
I'm not sure as I use Logic Apps to manage all my recurrence patterns. Also, I notice in your example that you specified an end date 1 day after the start date, which does not make sense in your scenario. — Joel Cochran, Jun 05 '23 at 15:50
Hi @JoelCochran, thanks again for reaching out. I think you're correct as regards not being able to achieve this with ADF. I have put a question out asking for help to achieve this with Logic Apps. Thanks — Patterson, Jun 05 '23 at 16:17

score 0 · Answer 1 · answered Jun 07 '23 at 12:38

You can also look at conditionals in order to achieve this requirement. The following is a demonstration for a pipeline (which will be scheduled to run every day) which executes the required pipeline on last but one day of every month.
I have taken 2 array parameters with pre-defined last but one day of each month in the format dd/MM, one parameter of the leap year case and another for normal year (As last but one day for Feb in leap year would be different):

last_but_one_ends: ["30/01","27/02","30/03","29/04","30/05","29/06","30/07","30/08","29/09","30/10","29/11","30/12"]

last_but_one_ends_leap_yr: ["30/01","28/02","30/03","29/04","30/05","29/06","30/07","30/08","29/09","30/10","29/11","30/12"]

Now, using set variable as an if condition, I check if its leap year or not (divisible by 4).

@if(equals(mod(int(formatDateTime(utcNow(),'yyyy')),4),0),'ok',0)

enter image description here

Now, based on the outcome of this, I have if condition activities that check the respective condition. It checks whether the current date (dd/MM format) is present in last_but_one_ends if its not a leap year and in last_but_one_ends_lear_yr if its a leap year.

enter image description here

Call the required pipeline when this above condition gives true and ignore the false case. The following is an debug output for today's date.

enter image description here

Schedule this pipeline to run daily. You can also schedule this pipeline to run only on the last 5 days in the schedule trigger settings as your required dates fall within this span.

enter image description here

Pipeline JSON (without trigger attached):

{
    "name": "pipeline2",
    "properties": {
        "activities": [
            {
                "name": "get day and month",
                "type": "SetVariable",
                "dependsOn": [],
                "userProperties": [],
                "typeProperties": {
                    "variableName": "date",
                    "value": {
                        "value": "@formatDateTime(utcnow(),'dd/MM')",
                        "type": "Expression"
                    }
                }
            },
            {
                "name": "if to check leap year",
                "type": "SetVariable",
                "dependsOn": [
                    {
                        "activity": "get day and month",
                        "dependencyConditions": [
                            "Succeeded"
                        ]
                    }
                ],
                "userProperties": [],
                "typeProperties": {
                    "variableName": "year",
                    "value": {
                        "value": "@if(equals(mod(int(formatDateTime(utcNow(),'yyyy')),4),0),'ok',0)",
                        "type": "Expression"
                    }
                }
            },
            {
                "name": "not leap year",
                "type": "IfCondition",
                "dependsOn": [
                    {
                        "activity": "if to check leap year",
                        "dependencyConditions": [
                            "Failed"
                        ]
                    }
                ],
                "userProperties": [],
                "typeProperties": {
                    "expression": {
                        "value": "@contains(pipeline().parameters.last_but_one_ends,variables('date'))",
                        "type": "Expression"
                    },
                    "ifTrueActivities": [
                        {
                            "name": "Execute Pipeline1",
                            "type": "ExecutePipeline",
                            "dependsOn": [],
                            "userProperties": [],
                            "typeProperties": {
                                "pipeline": {
                                    "referenceName": "pipeline1",
                                    "type": "PipelineReference"
                                },
                                "waitOnCompletion": true
                            }
                        }
                    ]
                }
            },
            {
                "name": "leap_year",
                "type": "IfCondition",
                "dependsOn": [
                    {
                        "activity": "if to check leap year",
                        "dependencyConditions": [
                            "Succeeded"
                        ]
                    }
                ],
                "userProperties": [],
                "typeProperties": {
                    "expression": {
                        "value": "@contains(pipeline().parameters.last_but_one_ends_leap_yr,variables('date'))",
                        "type": "Expression"
                    },
                    "ifTrueActivities": [
                        {
                            "name": "Execute Pipeline2",
                            "type": "ExecutePipeline",
                            "dependsOn": [],
                            "userProperties": [],
                            "typeProperties": {
                                "pipeline": {
                                    "referenceName": "pipeline1",
                                    "type": "PipelineReference"
                                },
                                "waitOnCompletion": true
                            }
                        }
                    ]
                }
            }
        ],
        "parameters": {
            "last_but_one_ends": {
                "type": "array",
                "defaultValue": [
                    "30/01",
                    "27/02",
                    "30/03",
                    "29/04",
                    "30/05",
                    "29/06",
                    "30/07",
                    "30/08",
                    "29/09",
                    "30/10",
                    "29/11",
                    "30/12"
                ]
            },
            "last_but_one_ends_leap_yr": {
                "type": "array",
                "defaultValue": [
                    "30/01",
                    "28/02",
                    "30/03",
                    "29/04",
                    "30/05",
                    "29/06",
                    "30/07",
                    "30/08",
                    "29/09",
                    "30/10",
                    "29/11",
                    "30/12"
                ]
            }
        },
        "variables": {
            "date": {
                "type": "String"
            },
            "year": {
                "type": "String"
            }
        },
        "annotations": []
    }
}

How to Configure Azure Data Factory with multiple schedules

1 Answers1