-1

We have the following infra structure to index application log data to ELK. filebeat -------> Logstash ------> Elastic search-----> kibana

All were working fine but suddenly Logstash server consume 99.9% CPU after which no indexing is happening. In filebeat we could see that "Error publishing events (retrying): EOF"

If we restart logstash service it starts indexing but when it reaches CPU 99.9%, it does not do anything.

Elastic search and kibana : AWS service Logstash : AWS Medium server Filebeat : AWS instance of our application test environment.

Please help us to resolve this issue.

Let me know if you need any other details.

Thanks in advance.

Sarathy
  • 442
  • 2
  • 9
  • 20
  • guess the size of document getting indexed on the Elasticsearch is huge. What is the heap being utilized in logstash? did you try changing heap size for logstash if it is excessively used? – DecKno Mar 10 '16 at 13:28

1 Answers1

0

Thanks Arivazhgan and daniel for your support and suggestions. I found the problem, the actual problem is my filter logic taking more time to process the log message. I have modified the log message format and optimized grok expression. Now everything working fine.

following changes have been done: 01. I have used mutate to convert few fields to int and float. This changes i did in pattern file itself. 02. I modified the log message format. 03. optimized the grok expression.

Sarathy
  • 442
  • 2
  • 9
  • 20