Using milestone 2 input plugin 'file'?

Question

I am reading a log file using Logstash. Here are the Files:

Config File:

input { 
file{
    path => "/home/cdot/Desktop/auth_log"
    start_position => beginning
}
}

filter{
grok{

match => ["message", "%{TIMESTAMP_ISO8601: timestamp} %{HOSTNAME: server-name} %{WORD: action}: %{WORD: machine}(%{GREEDYDATA: command}):%{GREEDYDATA:logline}"]
}
}

output {
    elasticsearch { host => localhost }
    stdout { codec => rubydebug }
}

Output:

Using milestone 2 input plugin 'file'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.1/plugin-milestones {:level=>:warn}

I am not getting any output. My log file has lines in the form of:

2014-05-09T04:02:08+05:30 bx920as1 runuser: pam_unix(runuser-l:session): session opened for user cyrus by (uid=0)
2014-05-26T04:02:09+05:30 bx920as1 runuser: pam_unix(runuser-l:session): session closed for user cyrus

Plz help.

EDIT:

After adding lines

start_position => beginning
    sincedb_path => "/dev/null"

to input I get the following output:

{
       "message" => "2014-05-26T04:02:09+05:30 bx920as1 runuser: pam_unix(runuser-l:session): session opened for user cyrus by (uid=0)",
      "@version" => "1",
    "@timestamp" => "2014-05-27T03:59:26.773Z",
          "host" => "cdot-HP-Pro-3330-MT",
          "path" => "/home/cdot/Desktop/auth_log",
       "logline" => " session opened for user cyrus by (uid=0)"
}
{
       "message" => "2014-05-26T04:02:09+05:30 bx920as1 runuser: pam_unix(runuser-l:session): session closed for user cyrus",
      "@version" => "1",
    "@timestamp" => "2014-05-27T03:59:26.774Z",
          "host" => "cdot-HP-Pro-3330-MT",
          "path" => "/home/cdot/Desktop/auth_log",
       "logline" => " session closed for user cyrus"
}
{
       "message" => "",
      "@version" => "1",
    "@timestamp" => "2014-05-27T03:59:26.774Z",
          "host" => "cdot-HP-Pro-3330-MT",
          "path" => "/home/cdot/Desktop/auth_log",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

Thus only the logline is getting captured and rest fields are not getting matched. Any idea?

score 2 · Answer 1 · answered May 27 '14 at 00:43

Logstash file input will keeps track of the current position of monitored log files and save the current position to sincedb which default path is your home directory. Please refer to here

So, the start_position => beginning only effect at the first time you start monitor the file. After that logstash will start from the position which save in sincedb.

So, if you always want to read the logs from the first line, add this config to your input file

sincedb_path => "/dev/null"

Or

Delete all the .sincedb files in your home directory. You can also input the logs to the monitor logs file after you start logstash.

Siddharth Trikha · Answer 2 · 2014-08-25T05:15:11.020

1

SOLVED: The problem was coming due to wrong expression for other identifiers (thus they were not getting displayed) and logline expression was right (thus was getting displayed).

edited Aug 25 '14 at 05:15

answered May 27 '14 at 04:34

Siddharth Trikha

2,648
8
57
101

Using milestone 2 input plugin 'file'?

2 Answers2