0

I'm creating a Data Factory pipeline with Web Activity, and that activity is using Azure Resource Graph API to query the resources table. And after I'll use copy data activity to copy the data to blob container. I already created it but the output file returns the same Web Activity results.

For example, and using the Web Activity,it is working well, but only returns 201 rows and if I execute the kusto query in Azure Graph Explorer returns 286 rows.

And if I filter by type = "microsoft.desktopvirtualization/hostpools" it will return 0 rows via API, but in Azure Graph Explorer returns 2 rows I think it will only queries the first 200 lines and not the whole set, because I tried with order by asc or desc and with "| take 500" (for example) and it didn't work either.

I checked the microsoft documentation and it says that the limit is 1000 rows, so I don't understand why it doesn't return more rows:

enter image description here

Results with type filter:

Azure Graph Explorer:

enter image description here

ADF Output:

enter image description here

ADF Pipeline:

URL: https://management.azure.com/providers/Microsoft.ResourceGraph/resources?api-version=2021-03-01

Method: POST

Body:

{
  "query": "resources"
}

Resource: https://management.azure.com

Headers: Content-Type; application/json enter image description here

Does anyone have similar issue/experience?

Thanks a lot!

coding
  • 135
  • 2
  • 9
  • Does your ADF (Managed Identity) have the access to ALL of the resources you are wanting to return. I would imagine it needs a reader role on the subscription, resource group, or whatever scope you are trying to do a web call for. – Trent Tamura May 10 '23 at 16:31
  • Hi @TrentTamura thanks for your reply. Since it is returning values ​​it is because ADF has access, otherwise it would give an error saying that it doesn't have permission or would not be authorised – coding May 10 '23 at 16:34
  • Well yes, but what I mean is my organization has thousands of Azure resources, but when I run this API call using Managed Identity, I only get 82 resources back. That is because my ADF only has at a minimum Reader Role to those 82 resources. It is bringing back resources from multiple subscriptions and resource groups, so it makes me believe it is only limited by the access my ADF has that is running the web request to other Azure resources. That make sense? – Trent Tamura May 10 '23 at 16:37
  • If you want to bring back ALL of the Azure Resources in your Tenant. You will have to ensure your ADF is a Reader on ALL resources within your tenant (All subscriptions, Resource Groups, etc) – Trent Tamura May 10 '23 at 16:39
  • I tested this and gave my ADF a Reader Role on another Resource Group that I had no Reader permissions on, and my Count went from 82 resources to 87 resources returning. So the access your ADF has to the resources definitely matters and will impact what returns in your API call – Trent Tamura May 10 '23 at 16:41
  • I think you're right yes @TrentTamura, because I executed this kusto query: resources | summarize count() by subscriptionId, and I notice that all the subscription returns the correct countrows but only one specific subscription is returning only one row, and I checked the roles, and my ADF is not there as Reader. And it is appearing one row because I have access to a specific resource of that subscription. I already asked devOps team to put the ADF as Reader and see again the results, but thanks a lot, really !!! – coding May 10 '23 at 16:59
  • I'll post this as the answer, please kindly mark as answer – Trent Tamura May 10 '23 at 17:45
  • Thanks a lot @TrentTamura, devOps team put the ADF as Reader and is working well !! – coding May 11 '23 at 07:24

1 Answers1

1

I would check the access your Azure Data Factory has. It needs to have Reader Role on all of the Azure resources you are expecting to return in the API Call:

Add Role Assignment

You can assign a role at a Resource, Resource Group, Subscription, or Management Group Level.

Cheers!

Trent Tamura
  • 1,079
  • 6
  • 16