2

We use swisscoms application cloud and are currently evaluating the new Elasticsearch service. We set it up including logstash and kibana.

We now added a user provided service to each of our apps that should use the common elasticsearch/logstash/kibana instance. When we first logged in into kibana we saw there was an index called logstash-, where all the logs of all applications go.

Now what we want is to have a index for each of the apps that writes to our elk instance. Lets say we have e apps (app1, app2, app3). We d like to have three indices (app1-..., app2-... and app3-...). Any ideas on how we can achieve that?

Is that a configuration that has to be done using ENV variables on Cloud foundry or is it something we have to configure within our Java and NodeJS apps

(app1-... , ...)?

Thanks in advance for your help.

Sufiyan Ghori
  • 18,164
  • 14
  • 82
  • 110
mooonli
  • 2,355
  • 4
  • 23
  • 32

1 Answers1

3

You can use Elasticsearch output plugin for logstash which is the recommended method of storing logs in Elasticsearch. This plugin has a configuration option called index which is used to define the name of the index to write events to. The default index name is logstash-%{+YYYY.MM.dd}

Use it along with if conditional to assign a name of the index for each app based on type, like this,

output {
  if [type] == "apache" {
    elasticsearch {
      index => "apache-website-index"
    }
  } elseif [type] == "nginx" {
      elasticsearch {
        index => "nginx-website-index"
    }
  }
}

Please have a look at this answer as well

Please comment if you have any question.

Sufiyan Ghori
  • 18,164
  • 14
  • 82
  • 110
  • Hi Sufyan Thanks for the feedback. I will try that out. What kind of properties do I have available there? Can I get Org, Space, App name and host? and is it possible to just prefix logstash-... with one of those instead of having a large if elseif construct in the end? – mooonli Jun 03 '18 at 05:20
  • We have elasticsearch, logstash and kibana on our appcloud instance. Each app uses a user provided service that connects to elasticsearch. – mooonli Jun 03 '18 at 05:49
  • how logstash collects logs from your app? you need to look at the template of your logstash that is used to collect logs from your app and send to elastic search. this will tell you the type of log, tag etc. or you can log at the log itsef in elasticsearch and check if there is a “type” field there. – Sufiyan Ghori Jun 03 '18 at 06:09
  • alright, I will try to find that out. Thanks a lot Sufiyan. – mooonli Jun 03 '18 at 06:15
  • if you haven't found it , you can assign the type to each app from your filebeat configurations under `/etc/filebeat/filebeat.yml`, please let me know if you have any further questions – Sufiyan Ghori Jun 08 '18 at 05:23