Sending JSON logs to AWS Cloudwatch - mostly it works great, but once in awhile, I may get a log entry that isn't quite pure JSON (or at least, oddly formatted). Here's an example of a single log entry from a Slack bot:
{"message": "Unhandled request ({'token': 'ezyBLAHBLAHBLAHDSDFL59', 'team_id': 'TF3BLAHBLAH', 'api_app_id': 'A01EBLAHBLAH', 'event': {'client_msg_id': '5ablahbd-blah-blah-blah-ffe18343blah', 'type': 'message', 'text': 'thanks', 'user': 'UFBLAHBLAH', 'ts': '1605733337.001300', 'team': 'TF3BLAHBLAH', 'blocks': [{'type': 'rich_text', 'block_id': 'gucN', 'elements': [{'type': 'rich_text_section', 'elements': [{'type': 'text', 'text': 'thanks'}]}]}], 'channel': 'D01BLAHBLAH', 'event_ts': '1605733337.001300', 'channel_type': 'im'}, 'type': 'event_callback', 'event_id': 'Ev0BLAHBLAH', 'event_time': 1605733337, 'authorizations': [{'enterprise_id': None, 'team_id': 'TFBLAHBLAH', 'user_id': 'U01BLAHBLAH', 'is_bot': True, 'is_enterprise_install': False}], 'is_ext_shared_channel': False, 'event_context': '1-message-TFBLAHBLAHV-D0BLAHBLAH'})", "level": "WARNING", "name": "slack_bolt.App", "time": "2020-11-18T21:08:18.184+00:00"}
So it is valid JSON, and Cloudwatch correctly parses what is there, but the bulk of the details of the unhandled request are trapped inside a string:
"message" : "Unhandled request(<lots_of_json>)"
"level": "WARNING"
"name": "slack_bolt.App"
"time": "2020-11-18T21:08:18.184+00:00"
What I WANT to get out of there is the <lots_of_json> part, and I want to have it interpreted as JSON - be able to report, sort, and aggregate on those fields, etc.
I can get about this far in a Cloudwatch Insights query:
fields @timestamp, @message
| filter message like 'Unhandled request'
| parse message 'Unhandled request (*)' as unhandled_payload
| sort @timestamp desc
| limit 20
And then this gives me the <lots_of_json> string in the ephemeral field unhandled_payload
Now how can I get that unhandled_payload JSON-formatted string parsed as JSON? The parse command only accepts globs or regexes and using either of those for this sounds... unpleasant. There must be a command to parse a JSON string, right? What is it?
("go fix the logging in the app" is not an acceptable answer for the purposes of this question)
