Unable to decode network response from AWS

Question

I'm trying to setup a websocket for AWS transcribe, but I've been stuck for over a day trying to decode the byte data / Uint8ArrayView that I receive as a response. Any help is very much appreciated, I've tried tons of decoding, ByteBuffers, etc, and can't figure it out.

Here is a little document on how the data should be coming in: https://docs.aws.amazon.com/transcribe/latest/dg/event-stream.html

Here is how somebody easily does this with the same response in Javascript: Using marshaller (Full code on GitHub)

let messageWrapper = eventStreamMarshaller.unmarshall(Buffer(message.data));
let messageBody = JSON.parse(String.fromCharCode.apply(String, messageWrapper.body));

And here is the data that I receive as a response:

[0, 0, 3, 212, 0, 0, 0, 103, 76, 21, 128, 165, 15, 58, 101, 120, 99, 101, 112, 116, 105, 111, 110, 45, 116, 121, 112, 101, 7, 0, 25, 73, 110, 118, 97, 108, 105, 100, 83, 105, 103, 110, 97, 116, 117, 114, 101, 69, 120, 99, 101, 112, 116, 105, 111, 110, 13, 58, 99, 111, 110, 116, 101, 110, 116, 45, 116, 121, 112, 101, 7, 0, 16, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 13, 58, 109, 101, 115, 115, 97, 103, 101, 45, 116, 121, 112, 101, 7, 0, 9, 101, 120, 99, 101, 112, 116, 105, 111, 110, 123, 34, 77, 101, 115, 115, 97, 103, 101, 34, 58, 34, 84, 104, 101, 32, 114, 101, 113, 117, 101, 115, 116, 32, 115, 105, 103, 110, 97, 116, 117, 114, 101, 32, 119, 101, 32, 99, 97, 108, 99, 117, 108, 97, 116, 101, 100, 32, 100, 111, 101, 115, 32, 110, 111, 116, 32, 109, 97, 116, 99, 104, 32, 116, 104, 101, 32, 115, 105, 103, 110, 97, 116, 117, 114, 101, 32, 121, 111, 117, 32, 112, 114, 111, 118, 105, 100, 101, 100, 46, 32, 67, 104, 101, 99, 107, 32, 121, 111, 117, 114, 32, 65, 87, 83, 32, 83, 101, 99, 114, 101, 116, 32, 65, 99, 99, 101, 115, 115, 32, 75, 101, 121, 32, 97, 110, 100, 32, 115, 105, 103, 110, 105, 110, 103, 32, 109, 101, 116, 104, 111, 100, 46, 32, 67, 111, 110, 115, 117, 108, 116, 32, 116, 104, 101, 32, 115, 101, 114, 118, 105, 99, 101, 32, 100, 111, 99, 117, 109, 101, 110, 116, 97, 116, 105, 111, 110, 32, 102, 111, 114, 32, 100, 101, 116, 97, 105, 108, 115, 46, 92, 110, 92, 110, 84, 104, 101, 32, 67, 97, 110, 111, 110, 105, 99, 97, 108, 32, 83, 116, 114, 105, 110, 103, 32, 102, 111, 114, 32, 116, 104, 105, 115, 32, 114, 101, 113, 117, 101, 115, 116, 32, 115, 104, 111, 117, 108, 100, 32, 104, 97, 118, 101, 32, 98, 101, 101, 110, 92, 110, 39, 71, 69, 84, 92, 110, 47, 115, 116, 114, 101, 97, 109, 45, 116, 114, 97, 110, 115, 99, 114, 105, 112, 116, 105, 111, 110, 45, 119, 101, 98, 115, 111, 99, 107, 101, 116, 92, 110, 88, 45, 65, 109, 122, 45, 65, 108, 103, 111, 114, 105, 116, 104, 109, 61, 65, 87, 83, 52, 45, 72, 77, 65, 67, 45, 83, 72, 65, 50, 53, 54, 38, 88, 45, 65, 109, 122, 45, 67, 114, 101, 100, 101, 110, 116, 105, 97, 108, 61, 65, 75, 73, 65, 84, 52, 55, 65, 90, 72, 82, 81, 79, 82, 88, 72, 77, 85, 77, 79, 37, 50, 70, 50, 48, 50, 49, 48, 52, 49, 48, 37, 50, 70, 117, 115, 45, 101, 97, 115, 116, 45, 50, 37, 50, 70, 116, 114, 97, 110, 115, 99, 114, 105, 98, 101, 37, 50, 70, 97, 119, 115, 52, 95, 114, 101, 113, 117, 101, 115, 116, 38, 88, 45, 65, 109, 122, 45, 68, 97, 116, 101, 61, 50, 48, 50, 49, 48, 52, 49, 48, 84, 49, 56, 48, 50, 52, 57, 90, 38, 88, 45, 65, 109, 122, 45, 69, 120, 112, 105, 114, 101, 115, 61, 56, 54, 52, 48, 48, 38, 88, 45, 65, 109, 122, 45, 83, 105, 103, 110, 101, 100, 72, 101, 97, 100, 101, 114, 115, 61, 104, 111, 115, 116, 38, 108, 97, 110, 103, 117, 97, 103, 101, 45, 99, 111, 100, 101, 61, 101, 110, 45, 85, 83, 38, 109, 101, 100, 105, 97, 45, 101, 110, 99, 111, 100, 105, 110, 103, 61, 112, 99, 109, 38, 115, 97, 109, 112, 108, 101, 45, 114, 97, 116, 101, 61, 49, 54, 48, 48, 48, 92, 110, 104, 111, 115, 116, 58, 116, 114, 97, 110, 115, 99, 114, 105, 98, 101, 115, 116, 114, 101, 97, 109, 105, 110, 103, 46, 117, 115, 45, 101, 97, 115, 116, 45, 50, 46, 97, 109, 97, 122, 111, 110, 97, 119, 115, 46, 99, 111, 109, 58, 56, 52, 52, 51, 92, 110, 92, 110, 104, 111, 115, 116, 92, 110, 101, 51, 98, 48, 99, 52, 52, 50, 57, 56, 102, 99, 49, 99, 49, 52, 57, 97, 102, 98, 102, 52, 99, 56, 57, 57, 54, 102, 98, 57, 50, 52, 50, 55, 97, 101, 52, 49, 101, 52, 54, 52, 57, 98, 57, 51, 52, 99, 97, 52, 57, 53, 57, 57, 49, 98, 55, 56, 53, 50, 98, 56, 53, 53, 39, 92, 110, 92, 110, 84, 104, 101, 32, 83, 116, 114, 105, 110, 103, 45, 116, 111, 45, 83, 105, 103, 110, 32, 115, 104, 111, 117, 108, 100, 32, 104, 97, 118, 101, 32, 98, 101, 101, 110, 92, 110, 39, 65, 87, 83, 52, 45, 72, 77, 65, 67, 45, 83, 72, 65, 50, 53, 54, 92, 110, 50, 48, 50, 49, 48, 52, 49, 48, 84, 49, 56, 48, 50, 52, 57, 90, 92, 110, 50, 48, 50, 49, 48, 52, 49, 48, 47, 117, 115, 45, 101, 97, 115, 116, 45, 50, 47, 116, 114, 97, 110, 115, 99, 114, 105, 98, 101, 47, 97, 119, 115, 52, 95, 114, 101, 113, 117, 101, 115, 116, 92, 110, 52, 57, 54, 98, 50, 57, 49, 57, 54, 100, 100, 101, 98, 100, 57, 55, 101, 53, 100, 56, 54, 56, 54, 98, 54, 54, 100, 97, 101, 102, 48, 97, 100, 98, 102, 56, 53, 49, 48, 54, 53, 54, 99, 53, 52, 102, 100, 99, 98, 98, 101, 101, 100, 98, 49, 50, 54, 101, 55, 100, 54, 51, 57, 99, 39, 92, 110, 34, 125, 25, 123, 100, 219]

The first link you provided described the format of the message you posted. You'll need to decode the values yourself looking at the header and reading the messages in the body. That said, the header decodes as a `InvalidSignatureException`, which suggests some code you didn't post is broken. — Anon Coward, Apr 10 '21 at 18:40
Thanks for the input, can you tell me how you were able to decode the header? I am working on GZIP decoding now, and the code for the request is likely wrong somewhere, but my main goal is to decode the response in hopes in helps me out. Currently I'm working on a GZIP error `FormatException: Invalid GZip Signature` — August Kimo, Apr 10 '21 at 18:44
I wrote a small [python script](https://pastebin.com/L3R6aq53) to decode the header. That said, in this case, simply putting this data in a hex viewer would have shown you the error — Anon Coward, Apr 10 '21 at 19:08
The access key ID was in the error message. You can of course delete it and create a new one if you'd like, but it's not the end of the world if it gets out. — Anon Coward, Apr 10 '21 at 19:27

score 1 · Accepted Answer · answered Apr 10 '21 at 20:14

Huge thanks to @Anon-Coward I converted his python code into flutter code which decodes the message and makes the headers and body readable. It's a bit sloppy, but here it is: (testVal is the response from Amazon which is in my question).

                        Uint8List sample = Uint8List.fromList(testVal);
                        ByteData bytes = sample.buffer.asByteData();

                        var totalLength = bytes.lengthInBytes; //980
                        var headersLength = bytes.getInt8(7); //103

                        var data = sample.sublist(12);
                        
                        var left = headersLength;

                        while(left > 0){
                          //get the header name
                          var nameLength = data[0];
                          data = data.sublist(1);
                          var name = utf8.decode(data.sublist(0, nameLength));
                          data = data.sublist(nameLength);
                          //get the value type
                          var valueType = data[0];
                          data = data.sublist(1);
                          //get the value
                          var tempLenList = data.sublist(0, 2);
                          var tempLenBytes = tempLenList.buffer.asByteData();
                          var valueLength = tempLenBytes.getUint16(0, Endian.big);
                          data = data.sublist(2);
                          var value = data.sublist(0, valueLength);
                          data = data.sublist(valueLength);
                          var valueFinal;
                          if(valueType == 7){
                            valueFinal = utf8.decode(value);
                          }

                          left -= (4 + nameLength + valueLength);
                          print(name+ '   ' + valueFinal);
                        }

                        var remainingData = utf8.decode(data.sublist(0, (data.lengthInBytes - 4)));
                        print(remainingData);

Unable to decode network response from AWS

1 Answers1