1

I am trying to use rest api to do pagination as it is just sending the first page in Azure ADF going to blob storage. I am currently using AbsoluteUrl and $['@odata.nextLink'] to get over all the pages, the issue is I am getting this error, I have used first used the token activity to get the token and then used it in copy activity where the source is rest api dataset with headers dynamically coming from token activity and then used pagination. Can you point me in the right direction on if this is the correct approach or am I missing something.

enter image description here

This is how the import schema looks like:

enter image description here

And the error after importing schema

enter image description here

This is how my rest api configuration look like:

enter image description here

And this is how my token all web activity looks like:

enter image description here

Edit 2:

This is how the output is for Web activity:

enter image description here

Including the part of the snip that missed the access token:

enter image description here

This is the output for Copy Activity when Pagination is on:

enter image description here

This is the setup of the pipeline:

enter image description here

StupendousEnzio
  • 35
  • 2
  • 8
  • yea you have the right approach, are you correctly using the access_token in the additional authorization header in CopyData activity source ? – KarthikBhyresh-MT Oct 25 '21 at 02:59
  • @KarthikBhyresh-MT Yes I am using access_token in additional authorization header in Copy activity. Key: Authorization and Value: Bearer @{activity('Web').output.access_token} – StupendousEnzio Oct 25 '21 at 04:19
  • @KarthikBhyresh-MT I am doing everything right, but still I am getting this error where for the next page it is still throwing this error "Invalid Token". I manually used the pagination next page url and concatenated the "&access_token=VVAihQP3Hg2atCSn" to the link and it show the page. I am wondering if it is possible to do it in pagination AbsoluteUrl using dynamic function? – StupendousEnzio Oct 25 '21 at 04:36
  • @KarthikBhyresh-MT When I try to import schema, I am getting the error invalid access token even when the token is active and referenced correctly. – StupendousEnzio Oct 25 '21 at 04:41
  • did you try concatenating for authorization header `@concat('Bearer', activity('Login').output.access_token)` – KarthikBhyresh-MT Oct 25 '21 at 04:43

1 Answers1

0

HttpStatusCode 401 indicates the authentication was not completed or failed due to invalid credentials. It maybe that the access token is missing in request from copy activity or not referenced properly or is expired. Make sure you already have the right access to this API.

Here is an example with basic configuration requirements:

  1. Get the access token

enter image description here

enter image description here

  1. Ensure you are able to reference it dynamically using Add dynamic content fields. Modify the reference with respect to the output you have received from earlier Login Activity.

    • Additional headers: Authorization: @concat('Bearer', activity('Login').output.access_token)
    • AbsoluteUrl: ${result_root}.{nextPageURL}

Here is the official doc on Pagination support refer the supported Key and value pairs.

If you are getting the access token correctly but still seeing the error, try to Import Schema in Mapping Settings of copy activity. And make sure the nextPageUrl or odata.nextLink in your case is mapped correctly.

enter image description here

Recheck $['@odata.nextLink'] , AbsoluteUrl value as:

$.rootElementName.CollectionOfItems.nextLinkURL

KarthikBhyresh-MT
  • 4,560
  • 2
  • 5
  • 12
  • To get the access token in the web activity, I used post method and I have used Contect-Type as the token url not application/json, is that correct? The rest is as you have mentioned. When I try to get the mapping import schema it gives invalid token error. – StupendousEnzio Oct 25 '21 at 04:53
  • I am using this one @concat('Bearer ', activity('Token').output.access_token) and it works. When I try to import schema it throws invalid token error. – StupendousEnzio Oct 25 '21 at 05:11
  • Great! it works. But technically import schema would just fetch the output from your previous web activity no matter what value you provide for parameters or additional headers given it was run successfully. Can you try publish all and preview data for source in copy activity – KarthikBhyresh-MT Oct 25 '21 at 05:14
  • My apologies, it works in the sense, I am not having issue with Additional headers, like Authorization and @concat('Bearer ', activity('Token').output.access_token) except when I try to use pagination rule, it still throws error as invalid token. The same thing happens when I try import schema. The additional expression token values comes up and then it throws invalid token. – StupendousEnzio Oct 25 '21 at 05:20
  • I have made changes to my original post to show how each screen looks like. – StupendousEnzio Oct 25 '21 at 05:32
  • from the snips I see, you have headers as context-type, it should be Content-Type – KarthikBhyresh-MT Oct 25 '21 at 05:39
  • And during import schema, just pass the token value, don't add `Bearer` as you are already using Concat() to form it – KarthikBhyresh-MT Oct 25 '21 at 05:41
  • When I run for a single page without pagination it has no problem, but when I set it to pagination, it says invalid token. – StupendousEnzio Oct 25 '21 at 05:41
  • Changed the context-type spell and also removed the Bearer from Import schema, but still the same, invalid token. This is the exact Authorization key @concat('Bearer ', activity('Token').output.access_token) and AbsoluteUrl $['@odata.nextLink'] – StupendousEnzio Oct 25 '21 at 05:46
  • `access_token` is an example and you should provide the exact `Property` refering the access token in the output result from web activity. Can you share a snip of the complete pipeline setup and web activity output you see – KarthikBhyresh-MT Oct 25 '21 at 05:48
  • For absolute URL can you try $odata.nextLink – KarthikBhyresh-MT Oct 25 '21 at 05:49
  • can you share output you find [here](https://i.stack.imgur.com/xotIU.png) and what exactly are you trying to achieve, for better understanding to suggest – KarthikBhyresh-MT Oct 25 '21 at 05:57
  • I see in your snip, you have set Authentication to `None` in web activity – KarthikBhyresh-MT Oct 25 '21 at 05:59
  • I have added the outputs for both the activities and also the pipeline. I am using post method for the token url, in that I have provided the credentials in the body and set authentication to none. – StupendousEnzio Oct 25 '21 at 06:13
  • oki this seems to be different. From the web activity output snip, we can see no property named `access_token` ! you are just getting the `refresh_token`. You sohuld actually use refresh token to get and access token. Are you refering any documentation – KarthikBhyresh-MT Oct 25 '21 at 06:26
  • I don't have such a server to exactly repro and show to you – KarthikBhyresh-MT Oct 25 '21 at 06:27
  • In rush I missed the top section. There is access token and other details present. I have updated a new snip with just the access token. – StupendousEnzio Oct 25 '21 at 06:31
  • You mentioned you have provided credentials in the body of the POST, what credentials are these. This 401error may occur, if you try to use a delegated access token granted to a personal Microsoft account, to access an API that only supports work or school accounts (organizational accounts). – KarthikBhyresh-MT Oct 25 '21 at 06:59
  • The link is https://-------/oauth2/v1/token and in the body is the request grant-type, clientid, usrnm and pw. I am using the link to get the token first. The oauth2/v1 is not present in the base url. – StupendousEnzio Oct 25 '21 at 07:05
  • If am not wrong you are trying this https://learn.microsoft.com/en-us/graph/auth-v2-user#3-get-a-token – KarthikBhyresh-MT Oct 25 '21 at 07:08
  • Yes, I am using the same thing with additional parameters grant type, clientid, username and password. – StupendousEnzio Oct 25 '21 at 07:11
  • found this Note: Calling Microsoft Graph from a standalone web API is not currently supported by the Microsoft identity platform endpoint. For this scenario, you need to use the Azure AD endpoint. – KarthikBhyresh-MT Oct 25 '21 at 07:11
  • I am not calling microsoft graph api though, it is a rest api for which I am using token link to get the token. Had to change the base url to anonymous since we are looking for authentication type as Oauth2 which is not present in Azure we have OAuth2Client and include the access token as header in copy function. – StupendousEnzio Oct 25 '21 at 07:15
  • oki so you have the relative url configured in rest linked service ? try to setup like explained [here](https://lytix.be/connecting-and-using-ms-graph-in-azure-data-factory/#:~:text=document%20for%20later.-,Azure%20Data%20Factory,-Once%20the%20previous) – KarthikBhyresh-MT Oct 25 '21 at 09:11
  • For the first method, it says GetAppRegSecrets with url as rest api, I am thinking on should I include the token url or base url? Or just the concat method for web url for token instead of writing in the body? – StupendousEnzio Oct 25 '21 at 10:41
  • Okay, I may have got myself confused, in the rest dataset I have used dynamic query @dataset.tables, should I change that to relativeurl? – StupendousEnzio Oct 25 '21 at 11:00
  • try relative URL – KarthikBhyresh-MT Oct 27 '21 at 08:32