5

I have messages which are like below, the following message is one of the messages (have so many JSON formats which are not at all related to this)

request body to the server {'sender': '65ddd20eac244AAe619383e4d8cb558834', 'message': 'hello'}

I would like to group of these messages based on sender (alphanumeric value) which is enclosed in JSON.

Rajeev Uppala
  • 205
  • 1
  • 3
  • 13
  • What do you want to do with the group? Count the number of items in each group? Something else? What's your expected output? – Harish KM Jun 25 '20 at 15:37
  • I wanted to know how many messages are coming from each user and also how many distinct users in selected interval. – Rajeev Uppala Jun 25 '20 at 15:39

2 Answers2

13

CloudWatch Logs Insights query:

fields @message |
filter @message like 'request body to the server' |
parse @message "'sender': '*', 'message'" as sender |
stats count(*) by sender

Query results:

-------------------------------------------------
|               sender               | count(*) |
|------------------------------------|----------|
| 65ddd20eac244AAe619383e4d8cb558834 |     4    |
| 55ddd20eac244AAe619383e4d8cb558834 |     3    |
-------------------------------------------------

Screenshot: enter image description here

Harish KM
  • 1,303
  • 7
  • 17
  • Thank you harish, **Date is 2020-06-25 Request from User: 5044fbb94f2a6200644797dd0210c787 , User name is: pareti, and message: i want to create a ticket** Can we extract this as well? I want user name from this line – Rajeev Uppala Jun 26 '20 at 09:04
  • 2
    I think `fields @timestamp, @message | filter user_name like "pareti"` should work. – Adiii Jun 26 '20 at 09:31
  • 1
    @RajeevUppala `fields @message | filter @message like 'Request from User' | parse @message 'User name is: *, and message' as username` – Harish KM Jun 26 '20 at 10:00
1

you can use filter.

fields @timestamp, @message
| filter @message like "65ddd20eac244AAe619383e4d8cb558834"
| sort @timestamp desc
| limit 20

it will filter all the messages limit to 20 that send by 65ddd20eac244AAe619383e4d8cb558834.

update:

suppose the JSON log formate is this

{
    "sender": "65ddd20eac244AAe619383e4d8cb558835",
    "message": "Hi"
}

Now I want to count number of messages from 65ddd20eac244AAe619383e4d8cb558835

how many messages are coming from each user?

so simple you can run the query

stats count(sender) by sender |
# To filter only message the contain sender, to avoid lambda default logs
filter @message like "sender"

enter image description here if you want to see messages as well then modify the query a bit

stats count(*) by sender, message |

filter @message like "sender"

Here @message refers to whole to index where message refer to the JSON object message.

enter image description here

count_distinct

Returns the number of unique values for the field. If the field has very high cardinality (contains many unique values), the value returned by count_distinct is just an approximation.

how many distinct users in the selected interval?

It will list distinct users in 3hr of interval

 stats count_distinct(sender) as distinct_sender by bin(3hr) as interval

enter image description here

Adiii
  • 54,482
  • 7
  • 145
  • 148
  • I am doing this already, I wanted to get only such value which is alphanumeric and results will be grouped with the same alphanumeric value – Rajeev Uppala Jun 25 '20 at 15:20
  • okay got it, updating answer that might help what you are looking for – Adiii Jun 26 '20 at 01:55