0

We want to find the missed utterance rate per day from Lex logs.

For example:

  • Day 1 - 10 total utterances, 1 missed utterance
  • Day 2 - 20 total utterances, 4 missed utterance
  • ...

We want to be able to plot (missed utterances/total utterances x 100) per day (essentially, %) for one week, however we also need to include Lambda exceptions as part of our "missed utterances" count.

How do we calculate the total & missed utterance count and then obtain a %?

Is this possible in CloudWatch Insight Logs?

Expectd output is a graph for 7 days that has the percentage of missed utterances+exceptions to total utterances for the day.

  1. <date 1> 1%
  2. <date 2> 4%
  3. ...

One query we tried is:

 fields @message
| filter @message like /Exception/ or missedUtterance=1
| stats count(*) as exceptionCount, count(@message) as messageCount by bin(1d)
| display exceptionCount, (exceptionCount/messageCount) * 100
| sort @timestamp desc
Ermiya Eskandary
  • 15,323
  • 3
  • 31
  • 44
Sharvari Nagesh
  • 293
  • 3
  • 17
  • What is the question? How to get the stats or how to group them by day? What CloudWatch queries have you tried? – Ermiya Eskandary Oct 12 '21 at 11:32
  • @ErmiyaEskandary, we want to do both group them by day and calculate % using stats. We tried: – Sharvari Nagesh Oct 12 '21 at 12:02
  • What have you tried so far? What queries? – Ermiya Eskandary Oct 12 '21 at 12:03
  • 1
    @ErmiyaEskandary, I have added a query we tried in the question. We are able to count message, and missed utterances in different queries but not in a single query. Also grouping per day on this is also not working. – Sharvari Nagesh Oct 12 '21 at 12:28
  • Thanks for adding! Do you have to have the exception filtering or can that be ignored? – Ermiya Eskandary Oct 12 '21 at 12:42
  • 1
    @ErmiyaEskandary, we have to have exception filtering. We need to filter both lex and lambda logs. lex for missed calls and lambda for exceptions. then we have to add total errors (which is missed utterances + exceptions) and divide it by total messages and this has to be done per day – Sharvari Nagesh Oct 12 '21 at 12:47
  • Thank you for clarifying - I've now added that to the question to make your question clear and focused - I'd also recommend reading [How do I ask a good question?](https://stackoverflow.com/help/how-to-ask) as well as [How to create a Minimal, Reproducible Example](https://stackoverflow.com/help/minimal-reproducible-example) to prevent future back and forth :) – Ermiya Eskandary Oct 12 '21 at 12:48
  • What is the expected output? Do you want to show the exception count too? What is the issue with the above query? What do you want to see in the output? exception+missed count vs total count ? or exception count separate to missed count separate to total count then a %? – Ermiya Eskandary Oct 12 '21 at 13:35
  • @ErmiyaEskandary, Thank you for editing the quesiton :-). expected output is a graph for 7 days which has the percentage of missed utterances for the day. For example, If i have to show you in data: 1%, 4% ... 2%. So here the percentage needs to be calculated as (Total no. of missed utterances for day1) / (Total no. of messages for day1) x 100. So the missed Utterance rate needs to be calculated per day – Sharvari Nagesh Oct 12 '21 at 13:53
  • Thank you - I'll add it in but it's great to add the clarifications into the question; I think I should be able to come up with something... – Ermiya Eskandary Oct 12 '21 at 14:06

2 Answers2

1

This is unfortunately not possible to do within CloudWatch Log Insights as you would need to have 2 filter & 2 stats commands.

One filter would be used for getting the total count & another for getting the exception + missed utterance count.

While you can filter after one another, you can't get the counts of the result of each filter as 2 stats commands are not supported from within Log Insights (yet).

The most you can do within CloudWatch is to create a dashboard (or 2 Log Insights) with the below queries and calculate the percentage yourself:

fields @message
| stats count(*) as totalUtteranceCount by bin(1d)

fields @message
| filter @message like /Exception/ or missedUtterance = 1
| stats count(*) as exceptionAndMissedUtteranceCount by bin(1d)

In an enterprise chatbot project that I was an engineer on, I configured logs to be exported to ElasticSearch (OpenSearch in AWS Console), which then opened a whole new world of data analysis & gave me the ability to run statistics like the above.

If this is a must, I would look to implementing a similar solution until AWS improves CloudWatch Log Insights or provides this statistic within Amazon Lex itself.

In the long run, I would go with the first option however Log Insights is not meant to be a full-blown data analysis tool & you'll need to carry out much more analysis on your data (missed utterances, intents etc.) anyway in order to be able to improve your bot.

Hopefully, something like this query works in the future!

fields @message
| stats count(*) as totalUtteranceCount by bin(1d)
| filter @message like /Exception/ or missedUtterance = 1
| stats count(*) as exceptionAndMissedUtteranceCount by bin(1d)
| display (exceptionAndMissedUtteranceCount/totalUtteranceCount) * 100
| sort @timestamp desc
Ermiya Eskandary
  • 15,323
  • 3
  • 31
  • 44
0

We could get it working using the below query:

fields strcontains(@message, 'Error') as ErrorMessage
|fields strcontains(@message, '"missedUtterance":true') as @missedUtterance
| stats sum(ErrorMessage)  as  ErrorMessagCount , sum(missedUtterance) as missedCount,
 count(@message) as messageCount , (((ErrorMessagCount) + (missedCount)) /messageCount * 100) by bin(1d)

Here, we are using strcontains instead of parse because if there are no missed utterance on a particular day, the calculation (ErrorMesageCount + missedCount)/messageCount * 100 was empty.

Answer is like:

enter image description here

Sharvari Nagesh
  • 293
  • 3
  • 17