0

I have Logstash reading from a log file status.log and sending the output to an Elasticsearch instance.

I want to clean the data in Elasticsearch, for that I'm executing curl -XDELETE 'http://localhost:9200/index_name/_all'. If I check through the head plugin, the data is gone.

If that was not enough I'm also cleaning the log file with echo "" > status.log.

When I execute the application again, the old data reappears in Elasticsearch but with an updated @timestamp. The data is not present again in the status.log. The new data is inserted correctly in Elasticsearch.

How can I get rid of the old data? Is it still stored in Elasticsearch or Logstash has some kind of cache?

Federico Nafria
  • 1,397
  • 14
  • 39
  • 1
    Logstash doesn't keep a copy. Deleting the data from Elasticsearch should be enough. The only idea that comes to mind is that you have a status.log.00 (etc) and your logstash pattern is matching that file as well. – Alain Collins Mar 22 '15 at 04:26

1 Answers1

1

assuming you are working with the file plugin. If you add the stdout plugin to your output section like so:

stdout { codec => rubydebug }

logstash will display each processed log element on the console. When using the file plugin every processed logmessage gets a path field telling you were logstash read the message from. maybe that helps you to find out where the messages are coming from...

markus
  • 1,631
  • 2
  • 17
  • 31