AWS - subscribe multiple lambda logs to one elasticsearch service

Question

I have two log groups generated by two different lambda. When I subscribe one log group to my elasticsearch service, it is working. However, when I add the other log group I have the following error in the log generated by cloudwatch :

"responseBody": "{\"took\":5,\"errors\":true,\"items\":[{\"index\":{\"_index\":\"cwl-2018.03.01\",\"_type\":\"/aws/lambda/lambda-1\",\"_id\":\"33894733850010958003644005072668130559385092091818016768\",\"status\":400,\"error\":
{\"type\":\"illegal_argument_exception\",\"reason\":\"Rejecting mapping update to [cwl-2018.03.01] as the final mapping would have more than 1 type: [/aws/lambda/lambda-1, /aws/lambda/lambda-2]\"}}}]}"

How can I resolve this, and still have both log group in my Elasticsearch service, and visualize all the logs ?

Thank you.

score 31 · Accepted Answer · answered Mar 07 '18 at 17:05

The problem is that ElasticSearch 6.0.0 made a change that allows indices to only contain a single mapping type. (https://www.elastic.co/guide/en/elasticsearch/reference/6.0/removal-of-types.html) I assume you are running an ElasticSearch service instance that is using version 6.0.

The default Lambda JS file if created through the AWS console sets the index type to the log group name. An example of the JS file is on this gist (https://gist.github.com/iMilnb/27726a5004c0d4dc3dba3de01c65c575)

Line 86: action.index._type = payload.logGroup;

I personally have a modified version of that script in use and changed that line to be:

action.index._type = 'cwl';

I have logs from various different log groups streaming through to the same ElasticSearch instance. It makes sense to have them all be the same type since they are all CloudWatch logs versus having the type be the log group name. The name is also set in the @log_group field so queries can use that for filtering.

In my case, I did the following:

Deploy modified Lambda
Reindex today's index (cwl-2018.03.07 for example) to change the type for old documents from <log group name> to cwl
Entries from different log groups will now coexist.

Awesome, yes I was running it with ElasticSearch 6.0.0. Thank you for your time and your well explained answer ! — DionysoSong, Mar 08 '18 at 07:52
Thanks for the awesome solution. AWS should give clarification on this at the time of cloud watch log subscribe. — Suraj Singh Rathore, Sep 05 '19 at 05:18
Anyone have more details on step #2 (Reindex today's index), like some instructions or documentation? — Segfault, Feb 09 '21 at 17:42
@Segfault That's referring to Elasticsearch's Reindex API: https://www.elastic.co/guide/en/elasticsearch/reference/6.8/docs-reindex.html You set a `source` and `dest` index and can set a new `type` value in the `dest` to re-index as a different type — davidstoker, Feb 09 '21 at 22:56

score 15 · Answer 2 · answered Nov 05 '18 at 02:02

You can also modify the generated Lambda code like below to make it work with multiple CW log groups. If the Lambda function can create different ES index for the different log streams coming under the same log groups, then we can avoid this problem. So, you need to find the Lambda function LogsToElasticsearch_<AWS-ES-DOMAIN-NAME>, then the function function transform(payload), and finally change the index name formation part like below.

    // index name format: cwl-YYYY.MM.DD
    //var indexName = [
        //'cwl-' + timestamp.getUTCFullYear(),              // year
        //('0' + (timestamp.getUTCMonth() + 1)).slice(-2),  // month
        //('0' + timestamp.getUTCDate()).slice(-2)          // day
    //].join('.');

    var indexName = [
        'cwl-' + payload.logGroup.toLowerCase().split('/').join('-') + '-' + timestamp.getUTCFullYear(),              // log group + year
        ('0' + (timestamp.getUTCMonth() + 1)).slice(-2),  // month
        ('0' + timestamp.getUTCDate()).slice(-2)          // day
    ].join('.');

score 0 · Answer 3 · answered May 05 '20 at 17:19

Is it possible to forward all the cloudwatch log groups to a single index in ES? Like having one index "rds-logs-* "to stream logs from all my available RDS instances. example: error logs, slow-query logs, general logs, etc., of all RDS instances, would be required to be pushed under the same index(rds-logs-*)?

I tried the above-mentioned code change, but it pushes only the last log group that I had configured.

From AWS: by default, only 1 log group can stream log data into ElasticSearch service. Attempting to stream two log groups at the same time will result in log data of one log group override the log data of the other log group.

Wanted to check if we have a work-around for the same.

AWS - subscribe multiple lambda logs to one elasticsearch service

3 Answers3