4

My rsyslog logs locally correctly, however I wanted to also receive the logs remotely, so I added the rule:

*.* @@myIP:5141

to the end of my rsyslog.conf

To receive the output, I'm running logstash with the configuration

input { tcp { port => 5141 } }
output { stdout {} }

Logstash expects UTF-8 encoding, however I get the error

Received an event that has a different character encoding than you configured

The messages themselves seem to be garbled, or a mix of encodings, for example:

\u0016\u0003\u0002\u0000V\u0001\u0000\u0000R\u0003\u0002S\xB1R\xAB5K\xF6\\\xB9\xB2\xB4\xB1\xAE0\t\u007F\xDF`5\xF6\u0015\xC8)H\xD7H\xCF+&\xD5T5\u0000\u0000$\u00003\u0000E\u00009\u0000\x88\u0000\u0016\u00002\u0000D\u00008\u0000\x87\u0000\u0013\u0000f\u0000/\u0000A\u00005\u0000\x84\u0000

Note some entries are \u00, while others are \x. There are even multiple backslashes.

I was wondering if I messed up the settings somehow, or if there is something between me and the server which is messing up the messages?

I have also tried using the syslog logstash input, which gives the same result

Another example:

\u0016\u0003\u0002\u0000V\u0001\u0000\u0000R\u0003\u0002S\xB1RiZ^\xC3\xD9\u001Cj\a\xD4\xE0\xECr\x8E\xAC\xF5\u001A\xB9+\u07B9\xE5\xF9\xA3''z\u0018}9\u0000\u0000$\u00003\u0000E\u00009\u0000\x88\u0000\u0016\u00002\u0000D\u00008\u0000\x87\u0000\u0013\u0000f\u0000/\u0000A\u00005\u0000\x84\u0000

EDIT: I found the source of my problem, and it was encryption related. Unfortunately I can't disclose what I did to fix it, suffice to say John Petrone's answer below is good start for similar problems that future readers may experience

Paradise
  • 1,408
  • 1
  • 14
  • 25

3 Answers3

7

So that magic string you're getting back that looks like broken encoding is actually the SSL Handshake request.

I suspect what you've done is (like I just did) misconfigured the tcp input in logstash. Specifically, I forgot to add the ssl_enable => true. So it was listening for normal TCP and got SSL Handshake and dutifully recorded it as garbage.

docwhat
  • 11,435
  • 6
  • 55
  • 54
1

The problem is that a syslog source that you are ingesting is sending data in non UTF-8 format which is causing problems with Logstash, as that is what it is expecting. You've basically got 3 courses of action:

  1. Have Rsyslog correct this for you: Use the Rsyslog mmutf8fix module to fix invalid UTF-8 sequences. http://www.rsyslog.com/doc/mmutf8fix.html

  2. Change Logstash to use a more appropriate charset: You can change the default charset for the plain codec: http://logstash.net/docs/1.4.2/codecs/plain . You will need to experiment a bit, I'd check here for a starting point. https://logstash.jira.com/browse/LOGSTASH-1047

  3. Change your source to output UTF-8: Not knowing the sources being collected by Rsyslog I can't comment on what it would take to make this change.

I'd start with option 1 and if that does not work move to option 2.

John Petrone
  • 26,943
  • 6
  • 63
  • 68
  • Thanks for the input, 1. (Available since: 7.5.4) Looks promising but I'm stuck on version 7.2.4 on the remote. Upgrading it has been on the books for ages so I might just do that now. 2. Changing the charset would be perfect if I knew what was coming out, but I'm going to play around and see what happens. 3. Looking at the encoding on the files that rsyslog puts out (using file -bi) they all claim to be utf-8. So I'm not sure if that's a conversion rsyslog does before writing to the file, or if its indicitive of the source – Paradise Jul 01 '14 at 09:49
  • I just had a thought, could it be possible that rsyslog is trying to encrypt the traffic somehow and it's being mis-interpreted on my side? – Paradise Jul 01 '14 at 09:54
  • seems more likely that it's an encoding issue – John Petrone Jul 01 '14 at 13:47
  • I'm going to accept this answer, as its the best course of action for future readers although it doesn't solve my specific errors – Paradise Jul 10 '14 at 13:47
0

based on @docwhat answer.

nano logstash/pipeline/logstash.conf
# Or
nano /path/to/logstash.conf
input {
 beats {
      port => 5000
      ssl => false
 }
  #tcp {
  #   port => 5000
  #}
}
mlibre
  • 2,460
  • 3
  • 23
  • 32