AWS SQS, How to calculate the MD5 message digest for message attributes

Question

I'm currently trying to figure out how to calculate the MD5 message digest for message attributes in AWS. I'm following the uri SQS message metadata > Calculating the MD5 message digest for message attributes

Although this seems straight forward I'm trying to get the hash of the following attribute

var messageAttributes = new Dictionary<string, MessageAttributeValue>
{
    {"UserName", new MessageAttributeValue {DataType ="String", StringValue = "Me"}}
};

I've sent this message and the MD5 response was 3a6071d47534e3e07414fea5046fc217

Trying to figure out the documentation I thought this should have done the trick:

private void CustomCalc()
{
    var verifyMessageAttributes = new List<byte>();
    verifyMessageAttributes.AddRange(EncodeString("UserName"));
    verifyMessageAttributes.AddRange(EncodeString("String"));
    verifyMessageAttributes.AddRange(EncodeString("Me"));
    var verifyMessageAttributesMd5 = GetMd5Hash(verifyMessageAttributes.ToArray());
}

private List<byte> EncodeString(string data)
{
    var result = new List<byte>();
    if (BitConverter.IsLittleEndian)
    {
        result.AddRange(BitConverter.GetBytes(data.Length).Reverse());
    }
    else
    {
        result.AddRange(BitConverter.GetBytes(data.Length));
    }
    result.AddRange(Encoding.UTF8.GetBytes(data));
    return result;

}
public static string GetMd5Hash(byte[] input)
{
    using (var md5Hash = MD5.Create())
    {
        // Convert the input string to a byte array and compute the hash.
        var dataBytes = md5Hash.ComputeHash(input);

        // Create a new string builder to collect the bytes and create a string.
        var sBuilder = new StringBuilder();

        // Loop through each byte of the hashed data and format each one as a hexadecimal string.
        foreach (var dataByte in dataBytes)
        {
            sBuilder.Append(dataByte.ToString("x2"));
        }

        // Return the hexadecimal string.
        return sBuilder.ToString();
    }
}

But I ended up with this cf886cdabbe5576c0ca9dc51871d10ae Does anyone knows where I'm going wrong. It can't be that hard I guess I just don't see it at the moment.

What about the 4 byte lengths shown in the picture in the link? — jdweng, Oct 28 '20 at 15:02
This was dealt with in the EncodeString method, but thanks for pointing this one out. This was at first one of the things I overlooked and misinterpreted. — Samyne, Oct 28 '20 at 15:22

score 2 · Accepted Answer · answered Oct 28 '20 at 15:17

You are almost there but missing one step:

Encode the transport type (String or Binary) of the value (1 byte).

Note The logical data types String and Number use the String transport type.

The logical data type Binary uses the Binary transport type.

For the String transport type, encode 1.

For the Binary transport type, encode 2.

So you need to append either 1 or 2 before value, to indicate transport type. In your case:

var verifyMessageAttributes = new List<byte>();
verifyMessageAttributes.AddRange(EncodeString("UserName"));
verifyMessageAttributes.AddRange(EncodeString("String"));
verifyMessageAttributes.Add(1); // < here
verifyMessageAttributes.AddRange(EncodeString("Me"));
var verifyMessageAttributesMd5 = GetMd5Hash(verifyMessageAttributes.ToArray());

Thank you so much, I feel so stupid for overlooking this one. The answer was in front of me the whole time. — Samyne, Oct 28 '20 at 15:21
Happens with everyone, and exactly the right thing to do in such case is ask other people to look to the code — Evk, Oct 28 '20 at 15:30

Jenny · Answer 2 · 2020-11-10T15:20:48.497

0

Here is stand-alone solution in Node.js:

const md5 = require('md5');

const SIZE_LENGTH = 4;
const TRANSPORT_FOR_TYPE_STRING_OR_NUMBER = 1;
const transportType1 = ['String', 'Number'];

module.exports = (messageAttributes) => {
  const buffers = [];
  const keys = Object.keys(messageAttributes).sort();

  keys.forEach((key) => {
    const { DataType, StringValue } = messageAttributes[key];

    const nameSize = Buffer.alloc(SIZE_LENGTH);
    nameSize.writeUInt32BE(key.length);

    const name = Buffer.alloc(key.length);
    name.write(key);

    const typeSize = Buffer.alloc(SIZE_LENGTH);
    typeSize.writeUInt32BE(DataType.length);

    const type = Buffer.alloc(DataType.length);
    type.write(DataType);

    const transport = Buffer.alloc(1);

    let valueSize;
    let value;
    if (transportType1.includes(DataType)) {
      transport.writeUInt8(TRANSPORT_FOR_TYPE_STRING_OR_NUMBER);
      valueSize = Buffer.alloc(SIZE_LENGTH);
      valueSize.writeUInt32BE(StringValue.length);

      value = Buffer.alloc(StringValue.length);
      value.write(StringValue);
    } else {
      throw new Error(
        'Not implemented: MessageAttributes with type Binary are not supported at the moment.'
      );
    }

    const buffer = Buffer.concat([nameSize, name, typeSize, type, transport, valueSize, value]);

    buffers.push(buffer);
  });

  return md5(Buffer.concat(buffers));
};

See more in the sqslite repo on GitHub

edited Nov 10 '20 at 15:20

answered Nov 05 '20 at 22:12

Jenny

445
2
6
23

I hope you can find your way with the C# code I've added in my original question. It contains all the methods that are required for encoding the message. – Samyne Nov 06 '20 at 09:44
@Samyne your code does not seem to have 4 bite truncation of the name, type and value. I am also unable to run your code, since its missing pieces. Can you tell me if your truncation looks the same as the one I have in screenshot? – Jenny Nov 06 '20 at 15:07
I think you did a wrong assumption. You should take the length of you name value. Thus SellerName has a length of 10 and the value 10 should be converted to a 4 byte value. Then the SellerName must also be converted to an UT8 byte array. This is what I'm doing the in the method EncodeString – Samyne Nov 06 '20 at 15:23
@Samyne I tried that, but it still seems off. I am allocating buffer and filling it with zeros. So starting with 0000 (4 bytes). Then injecting the length (i tried injecting at the begining and at the end) so fist 4 bytes looked like 1000 and then 0010 (as seen on screenshot). Pattern was repeated for all 3 values. Does the string construction seem right to you? – Jenny Nov 06 '20 at 16:49
Did you take into account the IsLittleEndian check like I did because this could also mess things up. I can't debug your code because my knowledge about JavaScript languages is not adequate. – Samyne Nov 10 '20 at 09:29
@Samyne i figured it out in the end, I will update my post shortly. I problem was with how Buffer writes things in JavaScript. It does use big endian. Out of curiosity, in what instance LE is used in your code, can you give me an example? – Jenny Nov 10 '20 at 15:14

AWS SQS, How to calculate the MD5 message digest for message attributes

2 Answers2