How to select only specific log from records with imfile module?

Question

We have a Logstash agent that writes its internal logs to a file in fixed format (a message written in a single row, here it's formatetd for readability):

{
    :timestamp => "2015-08-20T18:24:07.458000+0300",
    :message => "SIGINT received. Shutting down the pipeline.",
    :level => :warn
}

I have to make rsyslog read the file, get logs with :level => :error and send it somewhere. I used imfile module for it, here is the configuration:

module(load="imfile")

input(type="imfile"
        file="/path/to/log_file"
        tag="logstash:"
        statefile="/path/to/state_file"
        severity="error"
        ruleset="logstash_internal")

ruleset(name="logstash_internal"){
        # Some action goes here
         ...
}

How can I select only error records from log file, not all?

Why not use Logstash to send the entries in question where you want them to go? — GregL, Aug 21 '15 at 15:33
As far as I know Logstash can't write its internal logs somewhere other than stdout or a file. — Aliaxander, Aug 24 '15 at 07:42
No, but it can treat it's own log file just like any other log. — GregL, Aug 24 '15 at 11:21

score 0 · Answer 1 · answered Sep 10 '15 at 08:16

I see two options here:

the dirty-but-simple one: just throw an if ($msg contains "error") and hope there's no message containing "error" that isn't an actual error
the clean-but-more complicated one: parse the logs from Logstash (or at least the part until the level) with mmnormalize and then do a filter on the parsed level property. You'd use mmnormalize similarly to Logstash's grok, though the internals are much different

How to select only specific log from records with imfile module?

1 Answers1