1

How do I use AWS Cloudwatch Log Insights' replace function?

The docs do not give working examples.

Given logs which contain paths such as /api/lumberjack/123/axe/456/fashion

I am trying:

fields message
| parse message "path=* " as path
| fields replace(path, /[0123456789]+/, 'ID') as uniqpath
| stats count(*) by uniqpath

I expect results like:

uniqpath | count
/api/lumberjack/ID/axe/ID/fashion | 12
/api/lumberjack/ID/beardedness | 44

But instead it complains "Invalid arguments, received: (path) but expected: (str: string,searchValue: string,replaceValue: string)"

xxjjnn
  • 14,591
  • 19
  • 61
  • 94

4 Answers4

5

The replace function accepts fields as input for the first argument.

What is not supported is the second argument. You are passing a regex which is not recognized as a string.

I have not found a way to convert the regex to string. But at least you can pass the fieldname path for the first param. I have tested it changing the regex for a normal string.

Query:

fields @message
| parse @message "path=*" as path
| fields replace(path, 'lumberjack', 'ID') as uniqpath
| stats count(*) by uniqpath

Results:

Query results

OARP
  • 3,429
  • 1
  • 11
  • 20
2

You can't replace by regex in one step, which would be ideal. Instead, you must tease out a possible id, coalesce to give it a default value, then replace that explicit value with a placeholder. This will allow you to get endpoint counts regardless of specific record ids.

This simple version works for endpoints that have AT MOST one id:

filter path like /\/api\/v/
| parse path /(?<@id1>\/[0-9]+)(\/|$)/
| fields coalesce(@id1, "") as id1
| fields replace(path, id1, "/{id}") as unique_path
| stats count(*) as requests by unique_path
| sort requests desc
| display unique_path, requests

If your record ids are more complex, you can update the regex accordingly. I had to do this because some of our app's endpoints accept ids as hex strings, ie "0x04f5".

This next version will work for up to 2 ids. For more, you can see how to repeat the given pattern, but I didn't find a shortcut for n-count ids in a URL path.

filter path like /\/api\/v/
| parse path /(?<@id1>\/[0-9]+)(\/|$)/
| fields coalesce(@id1, "") as id1
| fields replace(path, id1, "/{id}") as unique_path1
| parse unique_path1 /(?<@id2>\/[0-9]+)(\/|$)/
| fields coalesce(@id2, "") as id2
| fields replace(unique_path1, id2, "/{id2}") as unique_path
| stats count(*) as requests by unique_path
| sort requests desc
| display unique_path, requests
Jaime Bellmyer
  • 23,051
  • 7
  • 53
  • 50
1

The error message is pretty self-explanatory. The replace function expects an input of type string for the first argument. You provided a fieldname path which is not acceptable.

EDIT: To my surprise, the replace function accepts path as the first argument, which is not mentioned in the doc. See Omar's answer above.

jellycsc
  • 10,904
  • 2
  • 15
  • 32
0

Haven't tested it but I've once made something similar to that so yours should be something like the following:

fields @message
| parse @message "path=* " as path
| parse path /(?<@endpt>(\/[0123456789]+\/?))/
| fields replace(path, coalesce(@endpt, ''), '/{id}/') as uniqpath
| stats count(*) by uniqpath
MxGutierrez
  • 21
  • 2
  • 2
  • i had better luck with the coalesce on the outside of the replace: | fields coalesce(replace(path, pathid, '{id}'), path) as resourcePath – Charles Esau Sep 20 '22 at 21:07