0

We setup the environment for logstash & elastic search for log analysis.My hardware environment is high end even though the performance of the logstash is slow.Now Our goal is to find the following area where it takes much time input,filter or output. So we decided to go with monitoring it.We have one option as of now is to go with Java thread dump analysis.Is this the only way or any other options available.

Conf file details

input {
   file {
      path => ["home/**/*.log"]
      start_position => beginning
  }

}

filter {}

output {
    stdout { codec => rubydebug }
    elasticsearch {}
}
SkaveRat
  • 2,124
  • 3
  • 18
  • 34
Devaraj
  • 25
  • 6

1 Answers1

0

Personally, maybe not the best way, I was testing my configuration by running it locally after removing the output section and using the generator plugin. Other input (files, tcp, whatever is feasible to use the exact data) is possible, too.

Then I was modifying the configuration as needed and run the following command to see the differences: time java -jar logstash.jar agent -f conffile

Out of experience the if statements as well as detailed parsing took a loot of resources and I could save some by optimizing them (I don't see much here in your case). I personally I would first try to run it with and without the ruby block to see if this eats a lot of resources (not familiar how the ruby part is spawn).

You can also check to run the jobs parallel (especially with multiple cores), by default logstash uses only one worker (or used to do that). Please be aware that in some cases multiple workers can have unexpected side effects (e.g. running statistics etc., don't see this here).

volker
  • 1,805
  • 15
  • 15
  • We lets try and reply you soon – Devaraj Jan 29 '15 at 09:23
  • do u know what is easiest way to calculate no events per second in logstash? – Devaraj Jan 29 '15 at 10:52
  • metrics. But there was a bug in the past where it failed if you use multiple threads. Else you can look in ES how many events where going in. For testing "time java..." was sufficient for my cases (and run them a few times in a row since other system tasks can slow down a test run). – volker Jan 29 '15 at 16:48
  • Thanks i try and let you know – Devaraj Jan 30 '15 at 08:00