Using Log4J with LogStash

Question

I'm new to LogStash. I have some logs written from a Java application in Log4J. I'm in the process of trying to get those logs into ElasticSearch. For the life of me, I can't seem to get it to work consistently. Currently, I'm using the following logstash configuration:

input {
  file {
    type => "log4j"
    path => "/home/ubuntu/logs/application.log"
  }
}
filter {
  grok {
    type => "log4j"
    add_tag => [ "ApplicationName" ]
    match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}  %{LOGLEVEL:level}" ]
  }
}
output {
  elasticsearch {
    protocol => "http"
    codec => "plain"
    host => "[myIpAddress]"
    port => "[myPort]"
  }
}

This configuration seems to be hit or miss. I'm not sure why. For instance, I have two messages. One works, and the other throws a parse failure. Yet, I'm not sure why. Here are the messages and their respective results:

Tags                   Message
------                 -------
["_grokparsefailure"]  2014-04-04 20:14:11,613 TRACE c.g.w.MyJavaClass [pool-2- 
                       thread-6] message was null from https://domain.com/id-1/env-
                       MethodName

["ApplicationName"]    2014-04-04 20:14:11,960 TRACE c.g.w.MyJavaClass [pool-2-
                       thread-4] message was null from https://domain.com/id-1/stable-
                       MethodName

The one with ["ApplicationName"] has my custom fields of timestamp and level. However, the entry with ["_grokparsefailure"] does NOT have my custom fields. The strange piece is, the logs are nearly identical as shown in the message column above. This is really confusing me, yet, I don't know how to figure out what the problem is or how to get beyond it. Does anyone know how how I can use import log4j logs into logstash and get the following fields consistently:

Log Level
Timestamp
Log message
Machine Name
Thread

Thank you for any help you can provide. Even if I can just the log level, timestamp, and log message, that would be a HUGE help. I sincerely appreciate it!

i can't see how your grok pattern ever matches. You have two spaces before the %{LOGLEVEL:level}". If you have to deal with a variable number of spaces, use \s+ instead of spaces to match one or more spaces. And +1 on the grok debugger answer — Alcanzar, Jun 18 '14 at 16:20

score 28 · Answer 1 · edited Jan 25 '17 at 11:37

28

I'd recommend using the log4j socket listener for logstash and the log4j socket appender.

Logstash conf:

input {
  log4j {
    mode => server
    host => "0.0.0.0"
    port => [logstash_port]
    type => "log4j"
  }
}
output {
  elasticsearch {
    protocol => "http"
    host => "[myIpAddress]"
    port => "[myPort]"
  }
}

log4j.properties:

log4j.rootLogger=[myAppender]
log4j.appender.[myAppender]=org.apache.log4j.net.SocketAppender
log4j.appender.[myAppender].port=[log4j_port]
log4j.appender.[myAppender].remoteHost=[logstash_host]

There's more info in the logstash docs for their log4j input: http://logstash.net/docs/1.4.2/inputs/log4j

edited Jan 25 '17 at 11:37

Fritz Duchardt

11,026
4
41
60

answered Oct 07 '14 at 00:50

ranxxerox

281
3
2

I would recommend the same as you, but right now, I'm starting having problems when the amount of messages to be sent are quite large. Keep in mind that log4j read the events from a TCP connection. – alicia Oct 27 '15 at 14:08
I think log4j_port in log4j.properties file should be logstash_port. – imesh Jun 16 '16 at 09:13
does `[logstash_host]` have to be localhost? do we have to specify the ip or can the hostname suffice? – AbtPst Aug 31 '16 at 13:38
2

This plugin is deprecated due to some security issues: [source](https://www.elastic.co/blog/log4j-input-logstash) – cdalxndr Aug 21 '19 at 09:24

score 1 · Answer 2 · edited Jun 30 '21 at 11:04

1

On my blog (edit: removed dead link) I described how to send JSON message(s) to the ElasticSearch and then parse it with GROK. [Click to see blog post with description and Java example][1] In the post you find description but also simple maven project with example (complete project on github).

Hope it helps you.

edited Jun 30 '21 at 11:04

marstran

26,413
5
61
67

answered Sep 30 '14 at 18:13

albgorski

107
3

score 1 · Answer 3 · edited Aug 13 '18 at 10:22

It looks like the SocketAppender solution that was used before is deprecated because of some security issue. Currently the recommended solution is to use log4j fileAppender and then pass the file through filebeat plugin to logstash and then filter. For more information you can refer the below links:

https://www.elastic.co/blog/log4j-input-logstash

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html

Using Log4J with LogStash

3 Answers3