0

I am trying to configure ELK to take logs form two different applications. To my understanding, the Elasticsearch index is analogous to a to a database. So my question is:

Is it advisable to configure separate indices for different applications and Why? What are the pros and cons from it being used as/for a centralized logging system?

The ES has an inbuilt functionality with the default logstash-* index that it creates an index on a daily basis depending on the timestamp that it receives from the logs. Now how do I create an index for my app so that it is named different, say App1-* that behaves exactly as the logstash-* index i.e. is gets created on a daily basis from the timestamp?

I checked the ES documentation on index APIs but couldn't find sufficient information for this for a custom index. Any pointers in this?

mathakoot
  • 1,692
  • 2
  • 14
  • 26

1 Answers1

1

The major con for me in using multiple indices is that each one (and the underlying shards) chews up HEAP, which limits the number of indexes you can have open at one time. If you combine the data into one index, it will take less memory to keep the data available.

As for the daily indices, elasticsearch will create any index when asked to do so. Logstash, in the elasticsearch{} output, allows you to specify the name of the index, which can contain static ("logstash-") and dynamic elements (date, fields from the event, etc).

Be aware that there is a mapping template that is applied to the "logstash-*" indices. If you need any of that functionality, you'll need to handle it yourself.

Alain Collins
  • 16,268
  • 2
  • 32
  • 55
  • Any other set back in terms of querying/visualizing on Kibana in case of separate indices, @Alain Collins? And I am quite aware of the mapping template that the logstash-* index uses but I wanted to know if it is at all possible to get such a wildcard based daily creation feature for a custom index. – mathakoot Jul 27 '15 at 19:30
  • In kibana, each visualization is tied to an index, so if you want to combine your data on one panel, they'd need to be in the same index. You can, of course, have other panels on the dashboard using other indexes. – Alain Collins Jul 27 '15 at 19:51
  • Since panels are no more a thing with Kibana 4.x, you mean you can add visualizations irrespective of their index pattern on the same dash board. Correct me if I'm wrong? – mathakoot Jul 27 '15 at 20:01
  • Feel free to use the terms "visualization" and "panel" interchangeably. – Alain Collins Jul 28 '15 at 13:42
  • Sure. Thanks! And Alain, you got to tell me where did you learn these things from man? I mean any references/materials online? I'd highly appreciate you sharing it! – mathakoot Jul 28 '15 at 13:44
  • My experience comes from several years of using the products, and gathering knowledge from other sources. The IRC channels are good, and the elastic.co discussion system is ok as a reference sometimes. – Alain Collins Jul 28 '15 at 13:48
  • Great! Also I have been using Logstash Forwarder. Does it seem a good idea to push logs on different ports for different applications? – mathakoot Jul 28 '15 at 19:33