3

I'm migrating from the Microsoft.Azure.DocumentDB.Core SDK to the new Microsoft.Azure.Cosmos SDK (version 3.2).

The older SDK returns the ContinuationToken in the ResponseContinuation property and looks like this: 

+RID:knNRALdSjOBBAQAAAAAAAA==#RT:1#TRC:10#RTD:fUTWI3DNPK3FpLhdCXlVBTMxMjouMTouMjlVMjg7MTQ7MTkvMTE6ODgyOFsA#ISV:2#IEO:65536#FPC:AgEAAAAKAAyBIwBA3L8PAAQ=

The new SDK returns it in the ContinuationToken property as a JSON string representing a "CompositeToken" object and looks like this:

[{"compositeToken":{"token":"+RID:xS0oAMD5rWYLAAAAAAAAAA==#RT:1#TRC:2#RTD:fUTWI3DNPK3FpLhdCXlVBTMxMjouMTouMzhVMTE7NTc7NDUvNjc2NDozNFsA#ISV:2#IEO:65551#FPC:AQkAAAAAAAAADQAAAAAAAAA=","range":{"min":"","max":"FF"}},"orderByItems":[{"item":"2019-09-27T00:46:34.5653923Z"}],"rid":"xS0oAMD5rWYLAAAAAAAAAA==","skipCount":0,"filter":null}]

When I pull out just the token:

+RID:xS0oAMD5rWYLAAAAAAAAAA==#RT:1#TRC:2#RTD:fUTWI3DNPK3FpLhdCXlVBTMxMjouMTouMzhVMTE7NTc7NDUvNjc2NDozNFsA#ISV:2#IEO:65551#FPC:AQkAAAAAAAAADQAAAAAAAAA=

I get the following error:

Invalid JSON in continuation token +RID:xS0oAMD5rWYLAAAAAAAAAA==#RT:1#TRC:2#RTD:z9path31ttPskBzzyJFdBTQ0NDQ0AA==#ISV:2#IEO:65551#FPC:AQkAAAAAAAAADQAAAAAAAAA= for OrderBy~Context, exception: Unexpected character encountered while parsing value: +. Path '', line 0, position 0.

But if I pass in the entire composite token it works. I'm a little confused since both SDKs are just warppers for using the CosmosDB service so why the difference? I'd like to just use the shorter token object if I can as it is much smaller and easier to manage when exposed as an API. The new CompositeToken is being encoded by Swagger and adds the / character in front of every quote like so:

[{/"compositeToken/":{/"token":/"+RID:xS0oAMD5rWYLAAAAAAAAAA==#RT:1#TRC:2#RTD:fUTWI3DNPK3FpLhdCXlVBTMxMjouMTouMzhVMTE7NTc7NDUvNjc2NDozNFsA#ISV:2#IEO:65551#FPC:AQkAAAAAAAAADQAAAAAAAAA=/",/"range/":{/"min/":/"/",/"max/":/"FF/"}},/"orderByItems/":[{/"item/":/"2019-09-27T00:46:34.5653923Z/"}],/"rid/":/"xS0oAMD5rWYLAAAAAAAAAA==/",/"skipCount/":0,/"filter/":null}]

Which means I have to decode the string properly in order to use it which seems like an additional burdon on API consumers. Am I missing something? There does not seem to be much documentation on what this CompositeToken does with the new SDK and why we have this change.

EDIT

I've found that if I add .Replace("\"", "'") to the returned ContinuationToken in the new SDK I now have a version of the token that my API consumers can use to pass in for the next page since the single quotes won't be automatically encoded. However I am not sure this is the cleanest way to handle it and wanted to get some opinions. Non of the samples in the GitHub project goes into this. I was also wondering what the additional properties such as "range" and "orderByItems" are used for in the composite token as it is not clearly documented.

EDIT 2

Turns out that replacing double quotes with single quotes works when I use Swagger UI but not when I use Postman. Very strange. The same exact call that works in Swagger (using the single quote version of the compositeToken) gives me the following error when I use Postman:

Response status code does not indicate success: 400 Substatus: 0 Reason: (Invalid JSON in continuation token [{'compositeToken':{'token':' RID:xS0oAMD5rWYLAAAAAAAAAA== for OrderBy~Context, exception: Unterminated string. Expected delimiter: '. Path '[0].compositeToken.token', line 1, position 58.).

EDIT 3

Here is how the raw continuationToken looks when it is returned by both SwaggerUI as well as Postman. As you can see the string has double quotes in it so the consumer gets the results with additional backslashes added in the string in order to escape the double quotes. Obviously just passing it back as is results in an invalid JSON error which requires that either the user or the service has to correct for it.

SwaggerUI

Postman

Is there a best practice for managing this? Was also thinking about extracting the just the "token" property and passing that back to the user (as this is what the previous SDK used) and then injecting it back into a CompositeToken string when passed back since it appears as though the remaining properties of the CompositeToken never change (Perhaps this is what the previous SDK did?) - but I am concerned this may break in the future if this format evolves.

EDIT 4

For reference, here is what the continuationToken looked like in the previous SDK:

PreviousSDK

EDIT 5

I figured out why the token that had the double quotes replaced with single quotes did not work in Postman. It seems there is a known bug where Postman strips out any characters after the first hash symbol on string parameters: https://github.com/postmanlabs/postman-app-support/issues/6023 since there are multiple "#" in the token this will always truncate the input when passed in via Postman as a querystring. Works fine when passed into the body.

INNVTV
  • 3,155
  • 7
  • 37
  • 71
  • Please see if my answer helps: https://stackoverflow.com/questions/51867914/azure-cosmosdb-continuation-token-structure/51902344#51902344. In short, you should treat continuation token as opaque and not try to infer its value. – Gaurav Mantri Sep 27 '19 at 17:24
  • Thanks Gaurav. I actually did read your post prior to my question. I am more curious about why the 2 SDKs are supplying different token styles and what the additional data in the composite token in the new SDK is for. I am also interested in best practices for giving the token to API consumers as it is odd that it is a string representation of a JSON object rather than a simple token as it was before. I also find it odd that the simple "token" value works in the previous SDK but not in the new one. Trying to get my head around what is under the hood! – INNVTV Sep 27 '19 at 20:20
  • The hood is open source :) https://github.com/Azure/azure-cosmos-dotnet-v3. As Gaurav mentioned, the SDKs are not just thin wrappers, they are more like thick clients, with a lot of logic for different cases and operations. It makes sense than a major version change, there are breaking changes (following semantic versioning). – Matias Quaranta Sep 27 '19 at 22:24
  • Thanks Matias. I have been looking through the code however I'm more interested in best practices. For example I just found that if I add .Replace("\"", "'") to the returned ContinuationToken in the new SDK I now have a version of the token that my API consumers can use to pass in for the next page. However I am not sure this is the cleanest way to handle it and wanted to get some opinions. Non of the samples in the GitHub project goes into this. I was also wondering what the additional properties such as "range" and "orderByItems" are used for as it is not clearly documented. – INNVTV Sep 28 '19 at 18:46
  • The Continuation Token is a string. You should be able to return the string as-is to your consumers and receive it back as a string (not a JSON) and pass it to the SDK. Are you not able to do this? Why do you need to parse it and escape it? – Matias Quaranta Sep 30 '19 at 15:22
  • Hi Matias. Check out EDIT 3. I added screenshots of how the string is represented in both Swagger and Postman. The additional backslashes being injected into the token is why I cannot use it as-is. – INNVTV Sep 30 '19 at 16:26
  • Best question and edits to clarify the answer. Struggled a bit with postman until I put the token on the body and it worked. Must also mention that I didn't edit the token. Copy paste as it on PostMan body and it worked – NoloMokgosi Apr 25 '21 at 10:31

1 Answers1

1

Here is what I have found. Depending on your query complexity your token (or composite token) may have additional properties or a completely different structure. For example in calls where you are using ORDERBY your token will look like this:

[{'compositeToken':{'token':'+RID:xS0oAMD5rWYLAAAAAAAAAA==#RT:1#TRC:2#RTD:fUTWI3DNPK3FpLhdCXlVBTMxMjouMTouMzhVMTE7NTc7NDUvNjc2NDozNFsA#ISV:2#IEO:65551#FPC:AQkAAAAAAAAADQAAAAAAAAA=','range':{'min':'','max':'FF'}},'orderByItems':[{'item':'2019-09-27T00:46:34.5653923Z'}],'rid':'xS0oAMD5rWYLAAAAAAAAAA==','skipCount':0,'filter':null}]

As you can see this includes a property for "orderByItems", "skipCount" and filter. Also notice that this is wrapped in a "compositeToken" object.

On other call where I only do a CONTAINS query (or no WHERE clause at all) the token looks much simpler and is not wrapped in a "compositeToken" object at all. It also does not include many of the properties above.

[{'token':'+RID:xS0oAMD5rWYNAAAAAAAAAA==#RT:1#TRC:1#ISV:2#IEO:65551#FPC:AQ0AAAAAAAAADQAAAAAAAAA=','range':{'min':'','max':'FF'}}]

The only property the two have in common are "token" and "range" which seems to be the bare minimum.

So yes, you do indeed have to maintain the integrity of the token as it is given to you in a string format and as-is.

I've resolved the issue with the escaped characters by doing a .Replace("\"", "'") on the string before returning it to the user. For Postman you have to pass the string back in the body since using a queryString will truncate at the "#" symbol (This is a known issue with Postman as mentioned in my original post).

Hope this helps someone who is as curious as I was about the new continuation token formats in the Microsoft.Azure.Cosmos SDK and how to use them in your client applications including OpenAPI/SwaggerUI and Postman.

INNVTV
  • 3,155
  • 7
  • 37
  • 71