How to create custom indices when you stream Cloudwatch logs to elastic search

Question

We are looking to use one Kibana dashboard, i.e. one elastic search domain for all our micro services, right now the only thing I see preventing that is when subscribing cloud watch logs to elastic search, it always creates indices of the same format cwl-*

How can we create custom names for indices when we stream cloud watch logs to elastic search? Should we switch to log stash to create custom names for indices and stream that to the amazon elastic search?

How are you pushing cloudwatch logs to elasticsearch directly? — Yuvraj Gupta, Mar 04 '16 at 03:46
We are using the AWS managed elasticsearch service, you can stream your log group to elastic search service by adding a subscription to your aws elasticsearch domain that you created — alexfvolk, Mar 04 '16 at 18:43

score 4 · Answer 1 · edited Jul 29 '17 at 03:41

4

Once you Start Streaming CWL to Amazon Elasticsearch Service the index is created in ES. Got to the new lamba function and find section and modify.

var indexName = [
        'cwl-' + timestamp.getUTCFullYear(),              // year
        ('0' + (timestamp.getUTCMonth() + 1)).slice(-2),  // month
        ('0' + timestamp.getUTCDate()).slice(-2)          // day
    ].join('.');

edited Jul 29 '17 at 03:41

Pang

9,564
146
81
122

answered Jul 28 '17 at 18:33

David W

41
1

score 3 · Accepted Answer · answered Mar 17 '16 at 11:24

3

It seems to be, AWS deploys a Lambda Function called "LogsToElasticsearch_YOUR-ES-CLUSTER-NAME" into your account. Maybe you can change this Function, because of the code is simple inline Javascript.

answered Mar 17 '16 at 11:24

Oliver Schönherr

46
1

How to create custom indices when you stream Cloudwatch logs to elastic search

2 Answers2