Logback to Logstash encoding error

Question

I am using Logback to send logs from my Grails application to an ELK stack.

My logback appender

appender("tcp", LogstashTcpSocketAppender) {
  destination = "localhost:5044"
  SSLConfiguration aSsl = new SSLConfiguration()
  aSsl.trustStore = new KeyStoreFactoryBean()
  aSsl.trustStore.location = "classpath:logstashTrustStore.jks"
  aSsl.trustStore.password = "test123"
  if(aSsl.trustStore instanceof LifeCycle)
    aSsl.trustStore.start()
  if(aSsl instanceof LifeCycle)
    aSsl.start()
  ssl = aSsl

  encoder(LogstashEncoder) {
    encoding = "UTF-8"
  }
}

My logstash input config is

input {
  tcp {
    port => 5044
    mode => "server"
    type => log4j
  }
}

When I start my application the logs are sent to the ELK stack, but I get the following warning

[2017-01-22T10:57:14,801][WARN ][logstash.codecs.line     ] Received an event that has a different character encoding than you configured. {:text=>"\\a\\xDCl\\x86\\xFEڐ\\xD1\\u0006\\x92J\\xBDMTLN\\xBF@\\x93\\x9B\\x8E\\xC1\\xE8&\\xF5|\\xF1\\xF4\\u0000\\u0000:\\xC0#\\xC0'\\u0000<\\xC0%\\xC0)\\u0000g\\u0000@\\xC0\\t\\xC0\\u0013\\u0000/\\xC0\\u0004\\xC0\\u000E\\u00003\\u00002\\xC0+\\xC0/\\u0000\\x9C\\xC0-\\xC01\\u0000\\x9E\\u0000\\xA2\\xC0\\b\\xC0\\u0012\\u0000", :expected_charset=>"UTF-8"}

The logs I can see in Kibana only contain messages like that, but I don't understand where the encoding error could come from, as I configured my log appender to use UTF-8 and Logstash seems to be expecting UTF-8.

What am I doing wrong?

score 0 · Answer 1 · answered Jan 23 '17 at 05:14

0

This is because the default charset is UTF-8 and you might need to set the correct charset within your input something like this:

input {
  tcp {
    port => 5044
    mode => "server"
    type => log4j
    codec => plain {
       charset => "ISO-8859-1"
    }
  }
}

The available charsets. You could have a look at this ticket for more. Hope it helps!

answered Jan 23 '17 at 05:14

Kulasangar

9,046
5
51
82

I am pretty sure, that my LogstashEncoder is sending the data in UTF-8 encoding, as I have explicitly the encoding for the encoder. I also used this encoder to write to a file and the encoding there is definitely UTF-8 – Tobi Jan 23 '17 at 14:49

Logback to Logstash encoding error

1 Answers1