0
const stream = "84mawgoBMRqXAQgAGpIBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNS4zMzk1MzBcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5OS9odW1pZGl0eVwiLCBcInZhbHVlXCI6IFwie1xcXCJyZWxhdGl2ZSBodW1pZGl0eVxcXCI6IDEzfVwifSIalgEIABqRASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDUuMzQxNTE5XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTEvY2FzaW5ndGVtcGVyYXR1cmVcIiwgXCJ2YWx1ZVwiOiBcIntcXFwiY2Vsc2l1c1xcXCI6IDEwfVwifSIalwEIABqSASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDUuMzQzNTM4XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTMvaHVtaWRpdHlcIiwgXCJ2YWx1ZVwiOiBcIntcXFwicmVsYXRpdmUgaHVtaWRpdHlcXFwiOiA5Mn1cIn0iGpMBCAAajgEie1wiZGF0ZXRpbWVcIjogXCIyMDIyLTEwLTA4IDEzOjU3OjA1LjM0NTc4NVwiLCBcInRvcGljXCI6IFwiZ3JlZW5mb3JnZS91dWlkL3V0aWxpdHk5L2NvMlwiLCBcInZhbHVlXCI6IFwie1xcXCJwYXJ0cyBwZXIgbWlsbGlvblxcXCI6IDQ0MX1cIn0iGpcBCAAakgEie1wiZGF0ZXRpbWVcIjogXCIyMDIyLTEwLTA4IDEzOjU3OjA1LjM1MjY1M1wiLCBcInRvcGljXCI6IFwiZ3JlZW5mb3JnZS91dWlkL3V0aWxpdHk5L2FpcnByZXNzdXJlXCIsIFwidmFsdWVcIjogXCJ7XFxcImhlY3RvIHBhc2NhbHNcXFwiOiA1NzF9XCJ9IhqXAQgAGpIBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNS4zNTU5NzVcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5MS9odW1pZGl0eVwiLCBcInZhbHVlXCI6IFwie1xcXCJyZWxhdGl2ZSBodW1pZGl0eVxcXCI6IDY5fVwifSIapgEIABqhASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDUuMzU4MTA5XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTYvY2FzaW5ndGVtcGVyYXR1cmVcIiwgXCJ2YWx1ZVwiOiBcIntcXFwiY2Vsc2l1c1xcXCI6IDEyLjA5NzU4NTcxODcyNjkzOH1cIn0iGpQBCAAajwEie1wiZGF0ZXRpbWVcIjogXCIyMDIyLTEwLTA4IDEzOjU3OjA1LjM2NjczN1wiLCBcInRvcGljXCI6IFwiZ3JlZW5mb3JnZS91dWlkL3V0aWxpdHkzL2NvMlwiLCBcInZhbHVlXCI6IFwie1xcXCJwYXJ0cyBwZXIgbWlsbGlvblxcXCI6IDE1NDR9XCJ9IhqYAQgAGpMBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNS4zNzE5MjZcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5Ni9haXJwcmVzc3VyZVwiLCBcInZhbHVlXCI6IFwie1xcXCJoZWN0byBwYXNjYWxzXFxcIjogMTEwMH1cIn0iGqYBCAAaoQEie1wiZGF0ZXRpbWVcIjogXCIyMDIyLTEwLTA4IDEzOjU3OjA1LjM3NDQ1M1wiLCBcInRvcGljXCI6IFwiZ3JlZW5mb3JnZS91dWlkL3V0aWxpdHk5L2Nhc2luZ3RlbXBlcmF0dXJlXCIsIFwidmFsdWVcIjogXCJ7XFxcImNlbHNpdXNcXFwiOiAwLjk0MTg2MTc4NDk5MTgxNzh9XCJ9IhqiAQgAGp0BIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNS4zNzk0MzRcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5Ni9haXJ0ZW1wZXJhdHVyZVwiLCBcInZhbHVlXCI6IFwie1xcXCJjZWxzaXVzXFxcIjogMTYuMTM3OTM2NjY0Njg4MTl9XCJ9IhqXAQgAGpIBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNS4zODY3ODVcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5MS9haXJwcmVzc3VyZVwiLCBcInZhbHVlXCI6IFwie1xcXCJoZWN0byBwYXNjYWxzXFxcIjogOTg1fVwifSIamAEIABqTASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDYuMzM2MDI5XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTMvYWlycHJlc3N1cmVcIiwgXCJ2YWx1ZVwiOiBcIntcXFwiaGVjdG8gcGFzY2Fsc1xcXCI6IDEwMzh9XCJ9IhqWAQgAGpEBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNi4zNDY3MDFcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5MS9jYXNpbmd0ZW1wZXJhdHVyZVwiLCBcInZhbHVlXCI6IFwie1xcXCJjZWxzaXVzXFxcIjogMTB9XCJ9IhqXAQgAGpIBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNi4zNTA2NzhcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5My9odW1pZGl0eVwiLCBcInZhbHVlXCI6IFwie1xcXCJyZWxhdGl2ZSBodW1pZGl0eVxcXCI6IDk0fVwifSIakwEIABqOASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDYuMzUzODU2XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTkvY28yXCIsIFwidmFsdWVcIjogXCJ7XFxcInBhcnRzIHBlciBtaWxsaW9uXFxcIjogNDM3fVwifSIamwEIABqWASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDYuMzU3MjUwXCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTkvaGlnaGxldmVsZmxvYXRzd2l0Y2hcIiwgXCJ2YWx1ZVwiOiBcIntcXFwiZW5hYmxlZFxcXCI6IHRydWV9XCJ9IhqmAQgAGqEBIntcImRhdGV0aW1lXCI6IFwiMjAyMi0xMC0wOCAxMzo1NzowNi4zNzEyMDRcIiwgXCJ0b3BpY1wiOiBcImdyZWVuZm9yZ2UvdXVpZC91dGlsaXR5Ni9jYXNpbmd0ZW1wZXJhdHVyZVwiLCBcInZhbHVlXCI6IFwie1xcXCJjZWxzaXVzXFxcIjogMTIuMDk3NTg1NzE4NzI2OTM4fVwifSIalAEIABqPASJ7XCJkYXRldGltZVwiOiBcIjIwMjItMTAtMDggMTM6NTc6MDYuMzkwMjg3XCIsIFwidG9waWNcIjogXCJncmVlbmZvcmdlL3V1aWQvdXRpbGl0eTYvY28yXCIsIFwidmFsdWVcIjogXCJ7XFxcInBhcnRzIHBlciBtaWxsaW9uXFxcIjogMTUzNH1cIn0iGqIBCAAanQEie1wiZGF0ZXRpbWVcIjogXCIyMDIyLTEwLTA4IDEzOjU3OjA2LjQwMjcyM1wiLCBcInRvcGljXCI6IFwiZ3JlZW5mb3JnZS91dWlkL3V0aWxpdHkzL2FpcnRlbXBlcmF0dXJlXCIsIFwidmFsdWVcIjogXCJ7XFxcImNlbHNpdXNcXFwiOiA1Ny4xMjgyMTE3MTk1MzEwN31cIn0io+JZCmdplVwO/VbJVxs5Dg=="
        
var payload = Buffer.from(stream, 'base64').toString('utf8');
console.log('Decoded payload: %s', payload);

This is a simple code example with what I'm receiving from my Kinesis record data. I've tried parsing and do some string manipulation without success - plus I'm not sure why I should do so in the first place.

Amazon are clear on base64 and utf-8 (octet-stream) - not sure what I'm missing.

Generating the Record.kinesis.data from a kafka producer in python:

kafka_producer = KafkaProducer( bootstrap_servers= "10.0.0.129:9092",
                                value_serializer=lambda v: json.dumps(v).encode('utf-8'))
data = {
                "datetime": str(datetime.now()),
                "topic": str(msg.topic),
                "value": str(msg.payload.decode()),
                "environment": "test-v1"
        }
kafka_producer.send("greenforge-events", data)
fneron
  • 1,057
  • 3
  • 15
  • 39
  • 3
    This isn't utf8. It includes utf8-encoded quoted-string-encoded-JSON records separated by a binary header (so it looks like it's JSON that's gone through a printed-quotable filter and then got packed into a binary format). You'll need to look at what's generating this and what format it's using. I don't see anything that seems Kinesis-specific. How are you getting this stream? – Rob Napier Oct 10 '22 at 02:39
  • @RobNapier added more details. – fneron Oct 10 '22 at 02:43
  • Wouldn't you need to decode this with KafkaConsumer (or kafka.HighLevelConsumer in Node)? How is this related to Kinesis? Kafka has its own encoding format. – Rob Napier Oct 10 '22 at 02:53
  • I think there's no issue with the code above... I believe now it has to reside on the kafka-kinesis connector I'm using. https://github.com/awslabs/kinesis-kafka-connector – fneron Oct 10 '22 at 03:02

0 Answers0