0

I'm having trouble troubleshooting this issue I'm running into when trying to run the sample packages for the framework.

For each infant package I get the following error:

Operation on target Throw Exception - Invalid Infant failed: Operation on target Raise Error failed: Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Worker pipeline [Wait 3] is not valid in target Orchestrator [adf-procfwk-rg-procfwk]',Source=,''Type=System.Data.SqlClient.SqlException,Message=Worker pipeline [Wait 3] is not valid in target Orchestrator [adf-procfwk-rg-procfwk],Source=.Net SqlClient Data Provider,SqlErrorNumber=50000,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=50000,State=1,Message=Worker pipeline [Wait 3] is not valid in target Orchestrator [adf-procfwk-rg-procfwk],},],'

Looks like the "Validate Pipeline" function is returning some peculiar data. I'm not much of a C# / App programmer so I'm struggling to determine what function is being called and how the database is returning the values in 'untitled4.png'

The pipelines metadata looks correct and the pipeline exists in the ADF so I'm a bit perplexed by the error. If anyone has any insights I'd greatly appreciate it. Metadata Database Configuration

ADF Error Screenshot

Pipeline exists screenshot

Output of 'Validate Pipeline'

live mode

resource names

mysterious extra SPN

feemurk
  • 87
  • 3
  • 11

1 Answers1

0

Have you published the 'Wait 3' pipeline? The screen shot shows your running in Debug in Git connected mode.

The framework can only trigger and interact with published Worker pipelines in the target Data Factory/Synapse instance.

Thanks

Paul Andrew
  • 3,233
  • 2
  • 17
  • 37
  • Hi Paul, Yes I believe so. When I switch to live mode I can see all the artifacts there. I've added another screenshot to the initial question showing the live mode screenshot if that helps. – feemurk May 18 '21 at 15:15
  • Is the ADF name correct in the metadata? In the screenshot from SSMS it looks like you've concatenated the ADF name with the Resource Group Name, in the 'OrchestratorName' attribute. – Paul Andrew May 18 '21 at 15:19
  • Yes the resources and group names all correct. I've attached another image to confirm. (did it that way to create uniqueness - kept running into names that were already reserved) – feemurk May 18 '21 at 20:22
  • I recommend double checking all your metadata then and permissions for the SPN used by the function hitting your ADF instance. – Paul Andrew May 20 '21 at 03:21
  • Hi Paul - Did some digging into the Functions and found that a dependency event to the "ValidatePipline" call requesting the pipeline: https://management.azure.com/subscriptions/9f7e7857-86b8-49bb-a527-cb564314af5c/resourceGroups/rg-procfwk/providers/Microsoft.DataFactory/factories/adf-procfwk-rg-procfwk/pipelines/Wait%202?api-version=2018-06-01 .. the pattern is the same for all stage 1 pipelines and all guids/rg-names/pipeline names are correct. (continued on next msg) – feemurk May 20 '21 at 04:27
  • Line 42 requires ResourceGroupName, Orchestratorname, PipelineName - all of which are correct. So I can only assume that the constructor is failing to authenticate properly and isn't creating the _adfManagementClient properly. Thoughts?!?! – feemurk May 20 '21 at 04:37
  • Yes, so check the permissions of the SPN your using. Does it have contributor access to the data factory? – Paul Andrew May 20 '21 at 05:22
  • Figured it out! turns out that there was another SPN with the same name that was created - it was the only one showing up when assigning permissions to the ADF resource. So the SPN in the metadata database wasn't permitted to connect to the ADF. I checked again and found the second SPN in the IAM role assignment and everything started working properly. – feemurk May 23 '21 at 07:32