Logstash Grok Pattern for Rails 4?

Question

Anyone have a Logstash pattern for Ruby on Rails 4 multiline logs?

I only have a pattern for Rails 3, which has a much different log structure:

RUUID \h{32}
# rails controller with action
RCONTROLLER (?<controller>[^#]+)#(?<action>\w+)
# this will often be the only line:
RAILS4HEAD (?m)Started %{WORD:verb} "%{URIPATHPARAM:request}" for % {IPORHOST:clientip} at (?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}$
# for some a strange reason, params are stripped of {} - not sure that's a good idea.
RPROCESSING \W*Processing by %{RCONTROLLER} as (?<format>\S+)(?:\W*Parameters: {%{DATA:params}}\W*)?
RAILS4FOOT Completed %{NUMBER:response}%{DATA} in %{NUMBER:totalms}ms %{GREEDYDATA}
RAILS4PROFILE (?:\(Views: %{NUMBER:viewms}ms \| ActiveRecord: %{NUMBER:activerecordms}ms|\(ActiveRecord: %{NUMBER:activerecordms}ms)?
# putting it all together
RAILS4 %{RAILS4HEAD}(?:%{RPROCESSING})?(?<context>(?:%{DATA}\n)*)(?:%{RAILS4FOOT})?

Rails 4 logs are now in the format, which includes a timestamp and what looks like is an ID (#).

I, [2016-01-26T23:21:44.581108 #27447]  INFO -- : Started GET "/login" for XXX.XXX.XXX.XXX at 2016-01-26 23:21:44 -0800

score 0 · Answer 1 · answered Sep 05 '16 at 08:36

I searched a lot for a rails4 grok pattern, no hope.

I found some alternatives which i'm using now, they might be helpful for people having the same problem.

Approach #1

First, add the lograge gem to your application. this gem will simplify the rails log to something that you can parse easily using grok.

Then you can use the following pattern

LOGRAGE %{WORD:method}%{SPACE}%{DATA}%{SPACE}action=%{WORD:controller}#%{WORD:action}%{SPACE}status=%{INT:status}%{SPACE}duration=%{NUMBER:duration}%{SPACE}view=%{NUMBER:view}(%{SPACE}db=%{NUMBER:db})?%{GREEDYDATA}

This method was mentioned in this article.

Approach #2

As another alternative, you can use the json filter as in the following article with slightly different lograge configurations.

filter {
  json {
    source => "short_message"
    remove_field => "short_message"
  }
}

Logstash Grok Pattern for Rails 4?

1 Answers1