Log4Net and ElasticSearch - Multiple Indexes

Question

I am using log4net with an ElasticsearchAppender. Now I would like to add another appender of ElasticSearch but with another index. What I did for now is:

<appender name="ElasticSearchAppender" type="log4net.ElasticSearch.ElasticSearchAppender, log4net.ElasticSearch">
  <layout type="log4net.Layout.PatternLayout,log4net">
    <param name="ConversionPattern" value="%date - %level - %message %property{label} %property{mstimeload} %property{applicationid} %property{page} 
           %property{ipclient} %property{browser} %property{browsersignature} %property{appversion} %property{sessionuniquecodetag} %property{globalcountertailsloaded} 
           %property{ipserveraddress}" />
  </layout>
  <connectionString value="Server=myip;Index=log;Port=9200;rolling=true"/>
  <lossy value="true" />
  <bufferSize value="100" />
  <evaluator type="log4net.Core.LevelEvaluator">
    <threshold value="INFO"/>
  </evaluator>
</appender>
<appender name="ElasticSearchAppenderClient" type="log4net.ElasticSearch.ElasticSearchAppender, log4net.ElasticSearch">
  <layout type="log4net.Layout.PatternLayout,log4net">
    <param name="ConversionPattern" value="%date - %level - %message %property{label} %property{applicationid} %property{ipclient} %property{browser} %property{browserversione} %property{browsersignature}"/>
  </layout>
  <connectionString value="Server=myip;Index=logclient;Port=9200;rolling=true"/>
  <lossy value="true" />
  <bufferSize value="100" />
  <evaluator type="log4net.Core.LevelEvaluator">
    <threshold value="INFO"/>
  </evaluator>
</appender>

Then on the server side I have

private static readonly log4net.ILog log = log4net.LogManager.GetLogger("ElasticSearchAppender");
private static readonly log4net.ILog logclient = log4net.LogManager.GetLogger("ElasticSearchAppenderClient");

The problem is when I add the values of the properties. because what I do is for example:

log4net.ThreadContext.Properties["label"] = label;
log4net.ThreadContext.Properties["ipclient"] = ipclient;
log4net.ThreadContext.Properties["applicationid"] = applicationid;
log4net.ThreadContext.Properties["page"] = page;
...
log.Info(ex);

for the first log and

 log4net.ThreadContext.Properties["label"] = label;
 log4net.ThreadContext.Properties["browser"] = browser;
 log4net.ThreadContext.Properties["browserversion"] = browserversion;
 ...
 logclient.Info("");

But then in the log with index logclient I find the properties of the index log and viceversa. I would like instead to have two different and separate log indexes with different properties for each index.

score 2 · Answer 1 · answered Jan 22 '16 at 16:42

It looks like you are logging directly from your app into Elasticsearch to a single index logclient.

You could create two named loggers with separate indices, and the use the loggers appropriately throughout your code.

You also might want to consider changing the type instead of the index.

Also you might want to consider using Logstash. While it is another service you'll be responsible for, it can bring a lot to the table such as routing logs to indices and types or ip to geo_hash conversion.

Lastly, I switched from Log4net to Logary over a year ago to improve the integration with Logstash/Elasticsearch and have been extremely satisfied.

score 0 · Accepted Answer · answered Jan 22 '16 at 22:05

I think I have found the solution. In the web.config I did like this:

<log4net>
<appender name="ElasticSearchAppender" type="log4net.ElasticSearch.ElasticSearchAppender, log4net.ElasticSearch">
    <layout type="log4net.Layout.PatternLayout,log4net">
        <param name="ConversionPattern" value="%date %level %c{1} %c{2}" />
    </layout>
    <connectionString value="Server=localhost;Index=log;Port=9200;rolling=true"/>
    <lossy value="false" />
    <evaluator type="log4net.Core.LevelEvaluator">
            <threshold value="ALL" />
    </evaluator>
    <bufferSize value="100" />
</appender>
<appender name="ElasticSearchAppenderClient" type="log4net.ElasticSearch.ElasticSearchAppender, log4net.ElasticSearch">
    <layout type="log4net.Layout.PatternLayout,log4net">
        <param name="ConversionPattern" value="%date %level %c{1} %c{3}" />
    </layout>
    <connectionString value="Server=localhost;Index=logclient;Port=9200;rolling=true"/>
    <lossy value="false" />
    <evaluator type="log4net.Core.LevelEvaluator">
            <threshold value="ALL" />
    </evaluator>
    <bufferSize value="100" />
</appender>
<root>
    <level value="ALL"/>
    <appender-ref ref="ElasticSearchAppender" />
</root>
<logger name="ElasticSearchAppenderClient" additivity="False"> <!-- additivity="False" VERY IMPORTANT!! -->
    <level value="ALL" />
    <appender-ref ref="ElasticSearchAppenderClient" />
</logger>
</log4net>

And then I have created two different classes one for each Index and then I pass this class to my log:

private static readonly ILog _log = LogManager.GetLogger("ElasticSearchAppender");
private static readonly ILog _logclient = LogManager.GetLogger("ElasticSearchAppenderClient");

Log thislog = new Log ();
thislog.label = "log";
thislog.mstime = 1;
_log.Error(thislog);

LogClient thislogc = new LogClient();
thislogc.label = "logclient";
thislogc.browser = 2;
_logclient.Error(thislogc);

Log4Net and ElasticSearch - Multiple Indexes

2 Answers2