Cloudwatch Logs Insights working with multiple @messages

Question

I have the following query with the following output:

Query:
filter @message like /A:|B:/ 

Output:
[INFO] 2020-07-28T09:20:48.406Z requestid A: [{'Delivery': OK, 'Entry': 12323 }]
[INFO] 2020-07-28T09:20:48.407Z requestid B: {'MyValue':0}

I would like to print ONLY the A message when in the B message 'MyValue' = 0. For the above example, I would have to have the following output

Output:
[INFO] 2020-07-28T09:20:48.406Z requestid A: [{'Delivery': OK, 'Entry': 12323 }]

For the next example

[INFO] 2020-07-28T09:20:48.406Z requestid A: [{'Delivery': OK, 'Entry': 12323 }]
[INFO] 2020-07-28T09:20:48.407Z requestid B: {'MyValue':12}

The output should be empty

I can't do something like this because I miss the A message:

filter @message like /A:|B:/ 
filter MyValue = 0

Any ideas?

No, I didn't. You should export your data and process it in another platform like excel — Andres Gardiol, Aug 03 '21 at 19:05
@OmkarKulkarni I posted a possible solution below, hopefully it will help in your case — YoniXw, Sep 14 '21 at 13:07

score 4 · Answer 1 · answered Sep 14 '21 at 12:57

If anyone still interested, there IS ways to get the first and last from grouping by a field. So if you can fit your data into pairs of messages, it might help.

For example, given API Gateway access log (each row is a @message):

2021-09-14T14:09:00.452+03:00   (01c53288-5d25-*******) Extended Request Id: ***************
2021-09-14T14:09:00.452+03:00   (01c53288-5d25-*******) Verifying Usage Plan for request: 01c53288-5d25-*******. API Key: API Stage: **************/dev
2021-09-14T14:09:00.454+03:00   (01c53288-5d25-*******) API Key authorized because method 'ANY /path/{proxy+}' does not require API Key. Request will not contribute to throttle or quota limits
2021-09-14T14:09:00.454+03:00   (01c53288-5d25-*******) Usage Plan check succeeded for API Key and API Stage **************/dev
2021-09-14T14:09:00.454+03:00   (01c53288-5d25-*******) Starting execution for request: 01c53288-5d25-*******
2021-09-14T14:09:00.454+03:00   (01c53288-5d25-*******) HTTP Method: GET, Resource Path: /path/json.json
2021-09-14T14:09:00.468+03:00   (01c53288-5d25-*******) Method completed with status: 304

We can get method, uri and return code from the last 2 rows. To do this, I parse the relevant data into params, and then get them by doing aggregation by request id (that i also parse)

The magic is: using stats likesortsFirst() and sortsLast() and grouping by @reqid. (AWS Docs

Note: IMO, don't use earliest() and latest() as they depend on built-in @timestamp and worked weird for me where 2 sequential messages had the same timestamp

So, for example, using this query:

filter @message like "Method"
| parse @message /\((?<@reqid>.*?)\) (.*?) (Method: (?<@method>.*?), )?(.*?:)* (?<@data>[^\ ]*)/ 
| sort @timestamp desc
| stats sortsFirst(@method) as @reqMethod, sortsFirst(@data) as @reqPath, sortsLast(@data) as @reqCode by @reqid
| limit 20

We would get the following desired output:

@reqid                                  @reqMethod   @reqPath    @reqCode
f42e2b44-b858-45cb-*****************    GET          /path-******.json  304
fecddb03-3804-4ff5-*****************    OPTIONS      /path-******.json  200
e8e47185-6280-4e1e-*****************    GET          /path-******.json  304
e4fa9a0c-6d75-4e26-*****************    GET          /path-******.json  304

**Another note**: Unfortunately, since we use aggregations, if one would like to add another aggregation like `count(*)`, it does not seem currently possible in aws logs insight, as it doesn't support 2 level aggregation. — YoniXw, Sep 14 '21 at 14:06

Cloudwatch Logs Insights working with multiple @messages

1 Answers1