1

I have several problems with my configuration file. My goal is to parse three types of logs (for the moment). Here they are :

[29/05/2020 07:41:51.354] - ih912865 - 10.107.119.121 - 93 - Transaction 7635 COMPLETED 318 ms wait time 3183 ms
[29/05/2020 10:30:01.318] - Process status database sync - us1salx08167.corpnet2.com:8400(#52279) (load 0 grace period 5 minutes) : current date 2020/02/02 21:30:01 update date 2020/02/02 21:29:58 old state OK new state OK
   31730  31626  464 10980020     52:25 /plw/modules/bin/Lx86_64/opx2-intranet.exe -I /plw/modules/bin/Lx86_64/opx2-intranet.dxl -H /plw/modules/bin/Lx86_64 -L /plw/PLW_PROD/modules/preload-intranet.ini -- plw-sysconsole -port 8400 -logdir /plw/PLW_PROD/httpdocs/admin/log/ -slaves 2

Two of these logs can be in slave files named intranet-2020-06-25-8401.log or intranet-2020-06-25-8400.log the last one is in a master file named intranet-2020-06-25-8402.log For my tests I simplified the architecture of my log files, so I have a Log-test folder in which I put a slave file and a master file.

![image|690x200, 75%](upload://np8buDnMxagdq3wGusk0uP2scOm.png)

In these files I only put the corresponding logs and a different log to be able to see how to manage this case.

Here is the content of a "slave" :

[29/05/2020 07:41:51.354] - ih912865 - 10.107.199.125 - 93 - Transaction 7635 COMPLETED 318 ms wait time 3183 ms
[29/05/2020 10:30:01.318] - Process status database sync - us1salx08167.corpnet2.com:8400(#52279) (load 0 grace period 5 minutes) : current date 2020/02/02 21:30:01 update date 2020/02/02 21:29:58 old state OK new state OK
[29/05/2020 13:49:20.635] - Main process - Transaction SYSTEM 105238-12 SQL done 1 ms

Here is the content of a "master" :

   31730  31626  464 10980020     52:25 /plw/modules/bin/Lx86_64/opx2-intranet.exe -I /plw/modules/bin/Lx86_64/opx2-intranet.dxl -H /plw/modules/bin/Lx86_64 -L /plw/PLW_PROD/modules/preload-intranet.ini -- plw-sysconsole -port 8400 -logdir /plw/PLW_PROD/httpdocs/admin/log/ -slaves 2
[26/06/2020 21:38:01.386] - Main process - Starting HTTP service on port 8402 (socket #<MULTIVALENT stream socket waiting for connection at */8402 @ #x1022d2ddbb2>)

Now that you have a better understanding of my environment and my purpose, here's the problem. When I launch my logstash configuration, I retrieve my data in kibana. But kibana shows me that each log has been treated as coming from a slave file while I also have a log coming from a master file which doesn't have the same processing.

For a better understanding here is my configuration file :

input {
    file { 
        path => "/home/mathis/Documents/**/intranet*.log"
        exclude =>"*8402.log"
        sincedb_path => '/dev/null'
        start_position => beginning
        type => "slave"
    }
    file { 
        path => "/home/mathis/Documents/**/intranet*8402.log"
        sincedb_path => '/dev/null'
        type => "master"
    }
}
filter {
    if [type] == "slave" {
        grok {
            match => { "message" => ["\[%{DATESTAMP:eventtime}\] \- %{USERNAME:user} \- %{IPV4:clientip} \- %{NUMBER} \- %{WORD} %{NUMBER:exectime} %{WORD} %{NUMBER:time} %{GREEDYDATA:data} %{NUMBER:waittime}","\[%{DATESTAMP:eventtime}\] \- Process status database sync \- %{WORD}\.%{WORD}\.%{WORD}\:%{NUMBER:slavenumb}\(\#%{NUMBER}\) \(load %{NUMBER:nbutilisateur} grace period 5 minutes\) %{GREEDYDATA}"] }
            remove_field => "message"
        }
    
    date {
                match => [ "eventtime", "dd/MM/YYYY HH:mm:ss.SSS" ]
            target => "@timestamp"
        }
    }
    if [type] == "master" {
        grok {
                match => {"message" => ["%{NUMBER}%{SPACE}%{NUMBER}%{SPACE}%{NUMBER}%{SPACE}%{NUMBER}%{SPACE}(?<starttime>((?!<[0-9])%{HOUR}:)?%{MINUTE}(?::%{SECOND})(?![0-9]))"]}
                remove_field => "message"
        }
            date {
                match => [ "starttime", "HH:mm:ss","mm:ss" ]
            }
    }
        
    
}
output {
    elasticsearch {
        hosts => "127.0.0.1:9200"
        index => "logstash-local3-%{+YYYY.MM.dd}"
    }
}

And now this is what kibana shows me: enter image description here

As you can see, the type field is slave for all logs but we can also observe that the logs of the slave file "intranet-2020-06-25-8401.log" are correctly parsed and that the line of added log that does not interest me has the field tags _grokparsefailure (the middle line in the picture).

The other problem is that the other logs (the first two lines on the image) are from a slave file (which is not true) according to kibana, so I guess they are processed in my first grok which would explain why they also have the _grokparsefailure tags field.

So I guess there are several errors in my input and filter part. I've been searching for a long time and doing a lot of testing, could you help me fix my config file please?

Arc
  • 11,143
  • 4
  • 52
  • 75
Mathis
  • 25
  • 6
  • you might want to add ``stdout{codec=>rubydebug}`` on output so it will show you more insight of what is going on. Then show what is happening on it – yuliansen Jul 23 '20 at 13:50
  • Thank you but the result doesn't tell me anything I don't already know unfortunately. – Mathis Jul 24 '20 at 08:28

0 Answers0