Use bash, Azure CLI and REST API to access CosmosDB - how to get token and hash right?

Question

My ultimate goal is to upload a document to an existing CosmosDB (SQL) instance using bash and Azure CLI. Trouble is: Azure CLI does not offer a command to modify documents.

To work around this I looked into the az rest command and was hoping to call into CosmosDB's REST API to complete the task, but after hours of unsuccessful trying, I'm always getting the error:

Unauthorized({"code":"Unauthorized","message":"The input authorization token can't serve the request. Please check that the expected payload is built as per the protocol, and check the key being used. Server used the following payload to sign: 'get\ndbs\n\nsat, 25 apr 2020 13:50:22 +0000\n\n'\r\nActivityId: ..., Microsoft.Azure.Documents.Common/2.10.0"})

To keep it simple for now, I'm trying to list all my databases using the REST API, as described in the docs and if that works, move on to actual document uploads.

I'm also trying to follow the instructions presented in the docs on how to generate the authorization header.

The request to get the list of DBs is using the format: GET https://{databaseaccount}.documents.azure.com/dbs

Here's my bash script with problems/questions highlighted.

Part 1: Get an access token - question: is this the right token to begin with?

masterKey=$(az cosmosdb keys list --name MYDBINSTANCENAME --query primaryMasterKey --output tsv)

Part 2: Generate payload to hash - all content must be lowercase

verb="get"
resourceType="dbs"
resourceLink="dbs"
now=$((date -uR) | tr '[A-Z]' '[a-z]')
payload="$verb\n$resourceType\n$resourceLink\n$now\n\\n"

Part 3: Hash the payload - issue: the result of this hash is different from what the sample code in C# builds. So either one must be wrong but both result in the same error message.

hashedPayload=$(printf $payload | openssl dgst -sha256 -hmac $masterKey -binary)

Part 4: Create the required authentication string and convert to base 64 - question: is the base 64 encoding required for use with az rest?

authString="type=master&ver=1.0&sig=$hashedPayload" | base64

Part 5: create the headers string. This is using JSON notation because the blank separated approach does not work, although the docs state it should.

headers="{\"x-ms-date\": \"$now\", \"x-ms-version\": \"2018-12-31\", \"x-ms-documentdb-isquery\": \"true\", \"Content-Type\": \"application/query+json\", \"Authorization\": \"$authString\"}"

Part 6: call the REST API

az rest --verbose -m get -u "https://MYDBINSTANCENAME.documents.azure.com:443/dbs" --headers $headers

Output:

Request URL: 'https://MYDBINSTANCENAME.documents.azure.com:443/dbs'
Request method: 'GET'
Request headers:
    'User-Agent': 'AZURECLI/2.4.0 (HOMEBREW)'
    'Accept-Encoding': 'gzip, deflate'
    'Accept': '*/*'
    'Connection': 'keep-alive'
    'x-ms-date': 'sat, 25 apr 2020 13:54:10 +0000'
    'x-ms-version': '2018-12-31'
    'x-ms-documentdb-isquery': 'true'
    'Content-Type': 'application/query+json'
    'Authorization': 'type%3dmaster%26ver%...'
    'x-ms-client-request-id': 'a55357fe-411c-4adf-9fd6-1a255e010cca'
    'CommandName': 'rest'
    'ParameterSetName': '--verbose -m -u --headers'
Request body:
None
Response status: 401
Response headers:
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json'
    'Content-Location': 'https://MYDBINSTANCENAME.documents.azure.com/dbs'
    'Server': 'Microsoft-HTTPAPI/2.0'
    'x-ms-activity-id': '9119f8bd-53d9-4a87-8aff-a887ec652fed'
    'Strict-Transport-Security': 'max-age=31536000'
    'x-ms-gatewayversion': 'version=2.10.0'
    'Date': 'Sat, 25 Apr 2020 13:54:11 GMT'
Response content:
{"code":"Unauthorized","message":"The input authorization token can't serve the request. Please check that the expected payload is built as per the protocol, and check the key being used. Server used the following payload to sign: 'get\ndbs\n\nsat, 25 apr 2020 13:54:10 +0000\n\n'\r\nActivityId: 9119f8bd-53d9-4a87-8aff-a887ec652fed, Microsoft.Azure.Documents.Common/2.10.0"}

Answer 1

Thanks for your support @Gaurav Mantri and @Mark Brown. Your comments helped avoid death by a million paper cuts :-)

I've started a repository where I collect Azure CLI bash scripts. Find it at https://github.com/Krumelur/AzureScripts

Let me answer my own question(s) and provide a script that reads the existing DBs in the CosmosDB instance.

This is starting an interactive login in the browser window.

az login

Specify what resource we want to access. URIs together with required parameter values can be found at: https://learn.microsoft.com/en-us/rest/api/cosmos-db/cosmosdb-resource-uri-syntax-for-rest

comsosDbInstanceName="YOUR INSTANCE NAME GOES HERE"
baseUrl="https://$comsosDbInstanceName.documents.azure.com/"
verb="get"
resourceType="dbs"
resourceLink="dbs"
resourceId=""

Get the CosmosDB's master key. We need this to get access. This is the same key that can be found on the portal in the "Keys" section of the CosmosDB instance. The primary key is what the REST API refers to as the "master" key.

masterKey=$(az cosmosdb keys list --name $comsosDbInstanceName --query primaryMasterKey --output tsv)
echo "Masterkey: $masterKey"

CosmosDB REST API requires a hashed authorization header: https://learn.microsoft.com/de-de/rest/api/cosmos-db/access-control-on-cosmosdb-resources#authorization-header

To get date in HTTP format, locale must be set to US. Otherwise day names would be localized (to German, for example).

HTTP format is not directly supported by bash. To make it work, set the current timezone to GMT. The time format looks like this: "mon, 27 apr 2020 09:46:58 gmt".

now=$(env LANG=en_US TZ=GMT date '+%a, %d %b %Y %T %Z')
echo "Date: " $now

Concat verb, resource type, resource ID and date in the expected format. REST API expects the signature to be lowercase.

The "little" problem I was not aware of: trailing newlines (\n) are always truncated when outputting a string. This would break the hash, because CosmosDB expects them to be there. That's why the two trailing newlines are appended back after the lowercase operation.

signature="$(printf "%s" "$verb\n$resourceType\n$resourceId\n$now" | tr '[A-Z]' '[a-z]')\n\n"
echo "Signature: $signature"

Calculate a hash of the signature using the primary key of the CosmosDB instance. See https://superuser.com/questions/1546027/what-is-the-openssl-equivalent-of-this-given-c-hashing-code/1546036 for details on why this is so tricky. The tl;dr; version: OpenSSL and the server implementation interpret the hashing key differently.

hexKey=$(printf "$masterKey" | base64 --decode | hexdump -v -e '/1 "%02x"')
echo "Hex key: " $hexKey
hashedSignature=$(printf "$signature" | openssl dgst -sha256 -mac hmac -macopt hexkey:$hexKey -binary | base64)
echo "Hashed signature: $hashedSignature"```

Hashed signature is expected to be URL encoded. But of course, there's no built-in way in bash to do that. Geez. Luckily, we're dealing with a base64 string. The only character that needs encoding is the equals sign which becomes "%3d".

Build the authorization header using the format "type={typeoftoken}&ver={tokenversion}&sig={hashsignature}"

authString="type=master&ver=1.0&sig=$hashedSignature"
echo "Auth string: $authString"

Auth string is expected to be URL encoded. But of course there's no built-in way in bash to do that. Geez. This is not a full base64 encoding but instead only changes the characters we may see: = -> %3d, & -> %26, + => %2b, / => %2f

urlEncodedAuthString=$(printf "$authString" | sed 's/=/%3d/g' | sed 's/&/%26/g' | sed 's/+/%2b/g' | sed 's/\//%2f/g')
echo "URL encoded auth string: $urlEncodedAuthString"

Make the API call by combining base URL and resource link.

url="$baseUrl$resourceLink"
echo "URL: $url"

Can use "az rest" command:

az rest --verbose -m $verb -u $url --headers x-ms-date="$now" x-ms-version=2018-12-31 x-ms-documentdb-isquery=true Content-Type=application/query+json Authorization=$urlEncodedAuthString --debug

Alternative: use cURL

curl --request $verb -H "x-ms-date: $now" -H "x-ms-version: 2018-12-31" -H "x-ms-documentdb-isquery: true" -H "Content-Type: application/query+json" -H "Authorization: $urlEncodedAuthString" $url