Cloudwatch Insights search in multiline logs

Question

Example log:

REQUEST-ID:123 Video id=444
REQUEST-ID:123 Request=error
REQUEST-ID:534 Video id=555
REQUEST-ID:534 Request=ok

Question, how to find Video id of all requests with an error?

Answer 1

Here is the official response from AWS support:

Currently, we do not have support for nested queries, and hence your use-case won't be achievable with Insights. However, we can confirm that we have a feature request existing at our end for this use-case. We don't have an ETA now on when this will be implemented. I hope you understand. Please keep a watch on below link for any new feature updates:

https://aws.amazon.com/new/

As a workaround, for now, you could download partial query results(Run the query -> Actions -> Export -> Download query results(CSV) and filter the results to identify the video ids. Please find below one sample query that can help you in this case:

fields @timestamp, @message
| parse @message 'REQUEST-ID:* Request=*' as REQUESTID,Request
| parse @message 'REQUEST-ID:* Video id=*' as REQUESTIDVID,Videoid
| DISPLAY REQUESTID,Request,REQUESTIDVID,Videoid

Answer 2

As a workaround, you can use the below code if you are capturing the details in the lambda logs and group based on the requestId (@requestId) or time. (i.e., bin(5s) or bin(10s)to get the clear result in a tabular format.

| parse @message 'Request=*' as Request
| parse @message 'Video id=*' as Videoid
| stats sortsFirst(Request) as RequestType, sortsFirst(Videoid) as VideoId by @requestId
| filter @message like /Request|Video id/
| sort @timestamp desc

(or)

| parse @message 'Request=*' as Request
| parse @message 'Video id=*' as Videoid
| stats sortsFirst(Request) as RequestType, sortsFirst(Videoid) as VideoId by bin(10s)
| filter @message like /Request|Video id/
| sort @timestamp desc

The result will look like below.

RequestType	VideoId	requestId
error	444	123
ok	555	534