How to print or store all skipped lines of a large log file in logstash

Question

Some times due incorrect pattern of some lines of logs inside complete log files are skipped by logstash.

It only sends all lines having pattern matched to grok syntax i have written.

I want to see which lines of logs are skipped or not read by logstash so that we can make that pattern correct.

I have added

output {

if "_grokparsefailure" in [tags] {

stdout { 

codec => rubydebug

}

}

else

 {
 }
 }

but it only prints the grok parse failure for lines read by logstash

So how to print or store all skipped lines in a output log file .

score 0 · Answer 1 · answered Mar 22 '19 at 13:27

You need to set a output inside the else block, it is empty so logstash has nothing to do when the tag _grokparsefailure is not present.

Try something like this:

output {
    if "_grokparsefailure" in [tags] {
        file { 
            path => "/tmp/grok-failed.log"
        }      
    } else {
        file { 
            path => "/tmp/grok-correct.log"
        }      
    }
}

This will save both messages to different files.

How to print or store all skipped lines of a large log file in logstash

1 Answers1