Remove new lines from Message and Exception

Question

I am using Serilog in an Asp dot net core application. I want my my log files to be human readable, but also be able to be parsed easily.

The problem that I have come across is that exceptions are logged with line breaks. Some Microsoft events have messages that include new lines.

I would like to be able to parse the log file with one event per line.

I could write my own implementation of ITextFormatter that replaces new lines with \r\n, but that we mean I would need to duplicate much of the logic in MessageTemplateTextFormatter and other classes.

Answer 1

After digging into this for a while, I was able to come up with an answer. Andy West's answer pointed me in the right direction.

There are two separate issues here: CRLFs in the message and CRLFs in the exception.

I was able to solve the message problem by changing "{Message}" to "{Message:j}" in the outputTemplate.

Changing the exception was a little trickier. I had to add an enricher:

class ExceptionEnricher : ILogEventEnricher
{
    public void Enrich(LogEvent logEvent, ILogEventPropertyFactory propertyFactory)
    {
        if (logEvent.Exception == null)
            return;

        var logEventProperty = propertyFactory.CreateProperty("EscapedException", logEvent.Exception.ToString().Replace("\r\n", "\\r\\n"));
        logEvent.AddPropertyIfAbsent(logEventProperty);
    }        
}

This adds and new property called EscapedException. This has to be added to the configuration with .Enrich.With().

Then I replaced "{Exception}" with "{EscapedException}" in the outputTemplate.

Answer 2

This technique will remove all CRLF. First a new ITextFormatter.

    public class RemoveCrLf : ITextFormatter
    {
        private const int DefaultWriteBuffer = 256;

        private readonly ITextFormatter _textFormatter;

        /// <summary>
        /// 
        /// </summary>
        /// <param name="textFormatter"></param>
        public RemoveCrLf(ITextFormatter textFormatter)
        {
            _textFormatter = textFormatter;
        }

        /// <summary>
        /// 
        /// </summary>
        /// <param name="logEvent"></param>
        /// <param name="output"></param>
        public void Format(LogEvent logEvent, TextWriter output)
        {
            var buffer = new StringWriter(new StringBuilder(DefaultWriteBuffer));

            _textFormatter.Format(logEvent, buffer);

            var logText = buffer.ToString();

            output.WriteLine(logText.Trim().Replace("\n","\\n").Replace("\r","\\r"));
            output.Flush();
        }
    }

Use it like this

configuration.WriteTo.Console(new RemoveCrLf(new MessageTemplateTextFormatter("[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj} {Exception}")));

Of course, customise the output template as needed.

Answer 3

In reference to the solution from @Andrew Radford, that solution was working for me only on windows, but not on linux docker environment as expected. Therefore I have enhanced the solution for messages as well as exceptions.

I have used Environment.NewLine for case matching in Regex which will pick the case based on the hosted environment.

Add following class for Messages.

public class MessageEnricher : ILogEventEnricher
{
    public void Enrich(LogEvent logEvent, ILogEventPropertyFactory propertyFactory)
    {
        if (logEvent.MessageTemplate == null)
            return;

        var logEventProperty = propertyFactory.CreateProperty("EscapedMessage", Regex.Replace(logEvent.MessageTemplate.ToString(), Environment.NewLine, "[Newline]"));
        logEvent.AddPropertyIfAbsent(logEventProperty);
    }
}

Use following class for Exceptions.

    public void Enrich(LogEvent logEvent, ILogEventPropertyFactory propertyFactory)
    {
        if (logEvent.Exception == null)
            return;

        var logEventProperty = propertyFactory.CreateProperty("EscapedException", Regex.Replace(logEvent.Exception.ToString(), Environment.NewLine, "[Newline]"));
        logEvent.AddPropertyIfAbsent(logEventProperty);
    }

Then add these both helper classes into program.cs

           Log.Logger = new LoggerConfiguration()
          .ReadFrom.Configuration(Configuration)
          .Enrich.With(new ExceptionEnricher())
          .Enrich.With(new MessageEnricher())
          .Enrich.FromLogContext()
          .CreateLogger();

Update the appsettings output template

 "outputTemplate": "{Timestamp:yyyy-MM-dd HH:mm:ss.fff} {Level:u4} {SourceContext:l} : {EscapedMessage}{NewLine}{EscapedException}{NewLine}"

Answer 4

You didn't specify how you are logging exceptions, but assuming you are using something like:

Log.Error("An error occurred: {Exception}", exception);

then the exception will be rendered using ToString(). In that case, couldn't you simply do this instead:

Log.Error("An error occurred: {Exception}", exception.ToString().Replace(System.Environment.NewLine, "\\r\\n"));

Of course, you could refactor this into a method (maybe an extension method) if you need to do it more than one place.