How to Log to Elastic Search by NLog or SeriLog with authentications

Question

We are currently use azure scale set (many VMs on one source group with load balance and one availability set), we used to use NLog to log our web app action and errors, but now we asked/needs to use Elastic Search and also use centralized log for all azure vm instances instead of file per each instance.

I am new to ES and LogStash concepts, Do I need to replace NLog with something else? and How I move to use ES and unify all logs in one (I think to make nlog store in azure storage table as unify results or do I needs to use LogStash or you prefer something else)?

What is the most logging that give support for .net core app on azure multi VMs as described above?

Any help please?

Answer 1

For NLog there is a target "NLog.Targets.ElasticSearch" (nuget) which uses the Elasticsearch.Net package.

Usage:

<nlog>
  <extensions>
    <add assembly="NLog.Targets.ElasticSearch"/>
  </extensions>
  <targets>
    <target name="elastic" xsi:type="BufferingWrapper" flushTimeout="5000">
      <target xsi:type="ElasticSearch"  
         requireAuth="true"
         username="myUserName"
         password="coolpassword"/>
    </target>
  </targets>
  <rules>
    <logger name="*" minlevel="Info" writeTo="elastic" />
  </rules>
</nlog>

Docs for the parameters: https://github.com/ReactiveMarkets/NLog.Targets.ElasticSearch/wiki

Please note:

• If you need to use Elasticsearch.Net 6 (equivalent to Elastic Search version 6 or later), you need NLog.Targets.ElasticSearch version 5.
• For Elasticsearch.Net 5 you need to use NLog.Targets.ElasticSearch 4

Answer 2

Since you want to log into ElasticSearch, it's best if you use a Logging framework that supports Structured Logging which for [NLog] this feature is in beta RTM.

You can Serilog as your Logging Framework which supports Structured Logging.

And there's also Serilog sink for ElasticSearch https://github.com/serilog/serilog-sinks-elasticsearch

You should add these nuget packages to your project :

Serilog
Serilog.Sinks.ElasticSearch  

This is a sample code for config of Serilog to sink to ElasticSearch

var logger = new LoggerConfiguration()
    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
    {
        ModifyConnectionSettings = x => x.SetBasicAuthentication(username, password);
    })
    .CreateLogger();

Answer 3

Many recommends that the application should not write directly to ElasticSearch, but should just write to local files.

Then have a service (Ex. FileBeat) to upload the contents of the log-files into ElasticSearch.

This will optimize network traffic to the ElasticSearch instance (bulk), and will ensure logging is not lost if problems with the network or ElasticSearch instance is restarted because of maintenance.

Answer 4

SetBasicAuthentication is not longer used. BasicAuthentication is the new thing you should use.

var logger = new LoggerConfiguration()
    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"))
    {
        ....
        ModifyConnectionSettings = x => x.BasicAuthentication("elasticUsername", "elasticPassword"),
    })
    .CreateLogger();

https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/configuration-options.html

Answer 5

If you want to use Nlog to write your logs into Elastic, you can follow this approach, first add the Auth settings into your target file:

requireAuth="true"
username="*******"
password="********"

After you added these attributes to the target tag in nlog.config everything will work well. But the other issue may be you face is the SSL connection problem. If your Elastic endpoint has SSL.

One solution to fix this issue is disabling certificate validation on Elastic endpoint. To achieve this, you need to add another attribute to your Elastic target.

DisableCertificateValidation="true"

Now your Elastic target in nlog.config file should be something like this:

<target xsi:type="ElasticSearch" 
    name="elastic"               
    index="MyService-${date:format=yyyy.MM.dd}" 
    layout ="MyService-|${longdate}|${event-properties:item=EventId_Id}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}|url: ${aspnet-request-url}|action: ${aspnet-mvc-action}" 
    includeAllProperties="true"
    requireAuth="true" 
    username="*******"
    password="*********" 
    uri="https://elasticSampleaddress.com:9200" />

For more detail, you can also read my article about this problem :Writing logs into Elastic with NLog , ELK, and .Net 5.0