Accessing Azure blob storage using bash, curl

Question

I am attempting to use the Azure blob storage service from a bash script using the REST API. I know it is possible to accomplish this using various other tools or languages, however I'd like to do it as a bash script.

The script below is an attempt to list the blobs in an Azure storage container.

This script results in an authentication error. The signing string and headers look correct based on the REST API (reference) documentation. I suspect the problem may be in juggling the various parts of the signing process.

Has anyone successfully used bash and curl to access cloud storage resources like Azure or other providers?

#!/bin/bash

# List the blobs in an Azure storage container.

echo "usage: ${0##*/} <storage-account-name> <container-name> <access-key>"

storage_account="$1"
container_name="$2"
access_key="$3"

blob_store_url="blob.core.windows.net"
authorization="SharedKey"

request_method="GET"
request_date=$(TZ=GMT date "+%a, %d %h %Y %H:%M:%S %Z")
storage_service_version="2011-08-18"

# HTTP Request headers
x_ms_date_h="x-ms-date:$request_date"
x_ms_version_h="x-ms-version:$storage_service_version"

# Build the signature string
canonicalized_headers="${x_ms_date_h}\n${x_ms_version_h}"
canonicalized_resource="/${storage_account}/${container_name}"

string_to_sign="${request_method}\n\n\n\n\n\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}\ncomp:list\nrestype:container"

# Decode the Base64 encoded access key, convert to Hex.
decoded_hex_key="$(echo -n $access_key | base64 -d -w0 | xxd -p -c256)"

# Create the HMAC signature for the Authorization header
signature=$(echo -n "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" | sed 's/^.*= //' | base64 -w0)

authorization_header="Authorization: $authorization $storage_account:$signature"

curl \
  -H "$x_ms_date_h" \
  -H "$x_ms_version_h" \
  -H "$authorization_header" \
  "https://${storage_account}.${blob_store_url}/${container_name}?restype=container&comp=list"

Update - The storage service error and the corresponding signing string that the script generated.

Following is what the storage service returns for the AuthenticationFailed error.

<?xml version="1.0" encoding="utf-8"?>
<Error>
  <Code>AuthenticationFailed</Code>
  <Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:27e6337e-52f3-4e85-98c7-2fabaacd9ebc
Time:2013-11-21T22:10:11.7029042Z</Message>
  <AuthenticationErrorDetail>The MAC signature found in the HTTP request
'OGYxYjk1MTFkYmNkMCgzN2YzODQwNzcyNiIyYTQxZDg0OWFjNGJiZDlmNWY5YzM1ZWQzMWViMGFjYTAyZDY4NAo='
is not the same as any computed signature. Server used following string to sign:
'GET

x-ms-date:Thu, 21 Nov 2013 22:10:11 GMT
x-ms-version:2011-08-18
/storage_account_name/storage_container
comp:list
restype:container'
  </AuthenticationErrorDetail>
</Error>

Next is the string_to_sign that the script generates.

GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Thu, 21 Nov 2013 22:10:11 GMT\nx-ms-version:2011-08-18\n/storage_account_name/storage_container\ncomp:list\nrestype:container

score 16 · Accepted Answer · answered Jun 03 '15 at 00:05

I was able to get it working. There were two things wrong with this code, the first, as Patrick Park noted, was replacing the echo -n with printf. The second was replacing the sed magic with the -binary option on openssl.

Compare the original:

signature=$(echo -n "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | sed 's/^.*= //' | base64 -w0)

with the fixed:

signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary |  base64 -w0)

The echo change is needed because echo -n will not convert the \n into actual newlines.

The -binary change is needed because even though you are stripping off the bad part, openssl was still outputting the signature in ascii-encoded-hex, not in binary. So after it was passed to base64, the result was the b64 encoded version of the hex representation, instead of the raw value.

Maybe you could help me on [this](https://stackoverflow.com/questions/41853666/storage-service-api-call)? — Munchkin, Jan 26 '17 at 08:31
I've modified this a bit and added support for downloads. I've pasted here: https://gist.github.com/jrwren/ff46f4ba177f042ccdc48c080c198f60 — jrwren, Apr 18 '17 at 22:10
decoded_hex_key="$(echo -n $access_key | base64 -d -w0 | xxd -p -c256)" This command gives me base64: invalid input. Is there a work around for this?Still getting the same error — SureshCS, Aug 08 '18 at 10:09

score 1 · Answer 2 · answered Nov 20 '13 at 18:06

1

Use Fiddler (or an equivalent on your platform) to intercept the call to Windows Azure Storage. On failure, this will show you the string that the Storage Service used to authenticate the call and you can compare this with the one you used.

answered Nov 20 '13 at 18:06

Neil Mackenzie

2,817
14
11

Thanks for the excellent suggestion. Doing this is difficult due to the environment I'm running in. I'll hang onto this suggestion if my debugging continues to be fruitless. – yozhik Nov 22 '13 at 20:55

score 0 · Answer 3 · answered Nov 21 '13 at 03:46

0

Looking at the REST API documentation and your code above, I believe there's an issue with the way you're constructing canonicalized_resource string. You're missing the query parameters in that string. Your canonicalized_resource string should be:

canonicalized_resource="/${storage_account}/${container_name}\ncomp:list\nrestype:container"

answered Nov 21 '13 at 03:46

Gaurav Mantri

128,066
12
206
241

The query parameters are being appended during the assignment of the `string_to_sign` variable. The authentication error returned from the storage service when I run the script includes the string that Azure used to create its Authorization signature. When I print the `string_to_sign` variable, it looks exactly like the string that Azure is returning in the error. – yozhik Nov 21 '13 at 16:12
Sorry, my bad! Didn't see that code earlier. Can you share the output of `string_to_sign` created by you and the one returned by Windows Azure. I'm pretty sure there must be some minor issue there. – Gaurav Mantri Nov 21 '13 at 16:19
Another thing to look for is how you decode your account key. Coming from C# world, we do `Convert.FromBase64String()` to get the account key in bytes. That could be another reason for authentication failure. – Gaurav Mantri Nov 21 '13 at 16:24
I added the output of `string_to_sign` to the original question. I also included the error XML that the storage service generates in response. – yozhik Nov 21 '13 at 22:35
I am most suspicious of any of the various parts of the process to decode the account key, create the signature, then re-encode. I think I might write a simple signature generator using another language, probably Java, and run the signature generation side-by-side to compare. – yozhik Nov 21 '13 at 22:38
To more specifically answer your question about decoding the account key. I use `base64 -d -W0` to decode the key. This outputs the key bytes which are then passed through a pipe to `xxd -p -c256` to directly hex encode the key bytes. This step is necessary because (AFAIK) I can't pass the bytes directly to openssl on the command line so I need to use the `hexkey` format. – yozhik Nov 21 '13 at 23:08

score 0 · Answer 4 · answered Dec 02 '13 at 02:04

It looks like openssl dgst does not generate proper HMAC for you. I wrote a simple program in C that does the following:

Takes base64-encoded key from the command line and decodes it into binary.
Reads string to sign from standard input.
Uses libcrypto HMAC() routine to generate the signature.
base64-encodes the signature and prints the result to standard output.

I then replaced openssl dgst pipeline in your script with the call to my program and it did the trick.

Please note that the output you are getting from Azure is XML-wrapped and base-64 encoded, so you'll need to come up with some sort of parsing/conversion code for it.

Just a note - I used wazproxy as my reference, you may want to check it out. Also Wireshark is your best friend when it comes to analyzing any traffic/protocols. — KiteRunner, Dec 04 '13 at 01:01

score 0 · Answer 5 · answered Feb 16 '15 at 13:05

0

use printf instead of echo (it works for me)

for example:

SIGNATURE=printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt hexkey:$HEXKEY -binary | base64 -w0

answered Feb 16 '15 at 13:05

Patrick Park

11
1

Accessing Azure blob storage using bash, curl

5 Answers5

Linked