How to execute a data factory pipeline with different set of parameters using powershell?

Question

I have a requirement to run the pipeline multiple times with different set of parameters? Can you please help how to iterate the execution for multiple set of parameters?

Below code isn't working:

$fileNames =  '"prepTable" = "test_1_ddl"', '"prepTable" = "test_2_ddl"'

$parameters = $fileNames | ForEach-Object {
Invoke-AzDataFactoryV2Pipeline -ResourceGroupName $(rg_nm) -DataFactoryName $(adf_nm) -PipelineName pl_common_Table_Creation_updated -Parameter $parameters }

Isn't working? What's happening? Can you call the pipeline once? Break down the problem to smaller pieces, and actually explain what the problem is. — Nick.Mc, Jun 03 '21 at 08:48
Thanks for the reply @Nick.McDermaid Getting below error: Cannot validate argument on parameter 'Parameter'. The argument is null or empty. Provide an argument that is not null or empty, and then try the command again. — user3647619, Jun 03 '21 at 09:08
You are using `$parameters` as variable that collects the result of the commands inside the orEach-Object loop. That same $parameters variable therefore is null **inside** the loop (`-Parameter $parameters`). The `-Parameter` needs a Hashtable, see [Invoke-AzDataFactoryV2Pipeline](https://learn.microsoft.com/en-us/powershell/module/az.datafactory/invoke-azdatafactoryv2pipeline?view=azps-6.0.0#parameters) — Theo, Jun 03 '21 at 09:23
I am inexperienced in powershell, how can I pass $parameters as Hastable? I want to run the pipeline twice for below parameters: '"prepTable" = "test_1_ddl"', '"prepTable" = "test_2_ddl"' — user3647619, Jun 03 '21 at 09:31

score 0 · Answer 1 · answered Jun 03 '21 at 14:48

You can create an array of Hashtables in PowerShell like this:

$prepTables =  @{'prepTable' = 'test_1_ddl'}, @{'prepTable' = 'test_2_ddl'}

Then you can use it in a loop like:

$prepTables =  @{'prepTable' = 'test_1_ddl'}, @{'prepTable' = 'test_2_ddl'}
$result = $prepTables | ForEach-Object {
    # the $_ Automatic variable represents a single item (hashtable) from the aray
    Invoke-AzDataFactoryV2Pipeline -ResourceGroupName $rg_nm -DataFactoryName $adf_nm -PipelineName 'pl_common_Table_Creation_updated' -Parameter $_
}

Personally, I hate those long code lines, so try using Splatting

$prepTables =  @{'prepTable' = 'test_1_ddl'}, @{'prepTable' = 'test_2_ddl'}
$result = $prepTables | ForEach-Object {
    $splatParams = @{
        ResourceGroupName = $rg_nm
        DataFactoryName   = $adf_nm
        PipelineName      = 'pl_common_Table_Creation_updated'
        Parameter         = $_   # the $_ Automatic variable represents a single item (hashtable) from the aray
    }
    
    Invoke-AzDataFactoryV2Pipeline @splatParams
}

^{P.S. I'm assuming here that your $(rg_nm) and $(adf_nm) are just placeholders for the real ResourceGroupName and DataFactoryName.
The code above uses variables for those.}

How to execute a data factory pipeline with different set of parameters using powershell?

1 Answers1