0

I have write the code like as following,

<Extension charconv>
    Module xm_charconv
    AutodetectCharsets utf-8, euc-jp, utf-16, utf-32, iso8859-2
</Extension>

And,

<Input sql-ERlogs>
    Module      im_file
    File 'C:\Program Files\Microsoft SQL Server\MSSQL11.SQL\MSSQL\Log\ER*'
    ReadFromLast TRUE
    Exec        convert_fields("AUTO", "utf-8"); if $raw_event == '' drop();
</Input>

I got the output like following image

enter image description here

If I expand one of the log I got original log like this,

enter image description here

Why is it in Unicode characters before expand?

Cœur
  • 37,241
  • 25
  • 195
  • 267
Gunaseelan
  • 14,415
  • 11
  • 80
  • 128
  • Looks like UTF-16le was passed to a JSON encoder expecting wide chars or UTF-8. The NUL bytes of the encoded text were encoded as `\u0000` because NULs can't be displayed and are otherwise "dangerous". – ikegami Dec 22 '14 at 04:47
  • Also I have tried with `convert_fields("AUTO", "utf-16")`. Then the message like `:message=>"Received an event that has a different character encoding than you configured."` @ikegami – Gunaseelan Dec 22 '14 at 04:52

1 Answers1

0

I had this same issue. Turns out "AUTO" doesn't work reliably. The NXLOG manual on MS SQLSRV hints at what the answer is; you must specify type, such as UTF-16LE.

Check the file contents with a hex editor and compare to MS's chart https://msdn.microsoft.com/en-us/library/windows/desktop/dd374101%28v=vs.85%29.aspx

Exec    $Event = convert($raw_event,"UTF-16LE","UTF-8"); to_json();
ag107093
  • 1
  • 2