6

I have an ELK stack locally hosted (v7.0) on a Windows IIS web server and the logs are not making it to the server. Server is running, I can reach the reserved URL and get back the generic json package saying Elasticsearch is running and I can log into Kibana just fine, there's just no logs to see.

I have a bufferBaseFilename set in the apps that are logging, and when I go to that location the logs are actually there, properly indexed and all. I'm wondering why it never gets synced back to the server? It seems like a connection issue, but all the network stuff checks out. I'm probably missing something simple. Any thoughts? Let me know if you need more information!

Hershizer33
  • 1,206
  • 2
  • 23
  • 46
  • 2
    Maybe you misconfigured the sink, so please provide Elasticsearch sink configuration. You could also try to use Serilog's SelfLog, there is a good chance that you'll see corresponding errors in it: https://github.com/serilog/serilog/wiki/Debugging-and-Diagnostics – Inok Jan 25 '20 at 17:04
  • Config is nodeUris: "http://log.company.pvt/[appname]/host" | indexFormat:"[appname]-prod" | templateName: "[appname]-prod" | minimumLogEventLevel: "Information" | bufferBaseFilename: "C:\Temp\[Appname] Temp Logs\". If you'd like to see the configuration code which uses these settings let me know and Ill edit the post with it. Thanks for the tip on Selflog, will try that out today! – Hershizer33 Jan 27 '20 at 14:03
  • @Inok Just tried out Selflog, Im getting `Received failed ElasticSearch shipping result 400: System.Collections.Generic.Dictionary'2[System.String,System.Object]. Failed payload` for every log call. Is that implying that it can't reach the server? – Hershizer33 Jan 27 '20 at 14:15
  • It looks like the server is reachable. Probably, there is an issue with the log entries, but there must be some data instead of `Dictionary'2` and after `Failed payload`. Do you use the latest version of `serilog.sinks.elasticsearch` package? If not, try to update it (8.0.1 at this moment), and also try to explicitly reference the latest version of `Elasticsearch.Net` package (7.5.1). It probably won't fix the issue, but it has to make the error in the SelfLog more informative. Also, clear buffer files or, which is preferable, disable buffering until the issue will be fixed. – Inok Jan 27 '20 at 20:49
  • @Inok There was more to the log message, just didn't want to break the char limit. It has the whole message in there. If it would be helpful to see let me know, would need to sanitize it first. Also I installed wireshark on the server and found something interesting, `[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]` on ACK packet from my client. – Hershizer33 Jan 27 '20 at 21:41

2 Answers2

3

A frequent source for this error is a malformed (template) request that does not match your ES version (e.g. contains deprecated fields). You could try to

  • use a preview version of the nuget package
  • set DetectElasticsearchVersion to true
  • set RegisterTemplateFailure to IndexAnyway

You can configure the sink like so:

var loggerConfig = new LoggerConfiguration()
    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(...) ){
        // ...
        DetectElasticsearchVersion = true,
        RegisterTemplateFailure = RegisterTemplateFailure.IndexAnyway
     });
Frederik Struck-Schøning
  • 12,981
  • 8
  • 59
  • 68
CaringDev
  • 8,391
  • 1
  • 24
  • 43
  • Thanks for the reply! I couldnt get the alpha package to work (updated to it, but when I run I get `FileLoadException: Could not load file or assembly 'Serilog.Sinks.Elasticsearch, Version=8.0.1.0, Culture=neutral, PublicKeyToken=24c2f752a8e58a10' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)`. For the other 2 options, where am I setting those values? In the sink configuration? – Hershizer33 Jan 28 '20 at 19:25
  • 1
    @Hershizer33 looks like you have other dependencies referencing a different version / not restored the packages correctly / a missing or wrong assembly redirect... could you share your project? For the config part: I updated my answer. – CaringDev Jan 30 '20 at 06:53
  • It turns out I had a few things wrong, needed to set it to 0 replicas since it was a single node server, and then adding `AutoRegisterTemplate = true` and `AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7` to the client logger config fixed it, though I'm not sure which one did it. Would `DetectElasticsearchVersion = true` had achived the same thing? It feels wrong to tie my logger config to a specific Elasticsearch version. – Hershizer33 Jan 30 '20 at 14:13
  • 1
    @Hershizer33 yes, auto-detect should have fixed that as well. For the ES sink it was a design decision to support multiple ES versions. This implies the need to include it in the config. The ES team itself chose a different way: you have to use a different nuget package for every major version. Either way: updating your ES stack means touching the code ‍♂️ – CaringDev Jan 30 '20 at 14:24
  • I see, so which combo do you recommend, `AutoRegisterTemplate = true` and `DetectElasticsearchVersion = true`? At least then I'd only have to update the nuget package instead of doing that plus changing the version each upgrade? – Hershizer33 Jan 30 '20 at 14:28
  • 1
    I opted for having `Detect... = true` and `...Version = ` but your suggestion seems valid too. – CaringDev Jan 30 '20 at 15:04
  • 1
    Marking this as the answer even though it was solved in the comments. Thanks! – Hershizer33 Jan 30 '20 at 18:08
1

I had this issue and for me, it was the w3wp.exe process that blocked a couple earlier buffer logs from pushing to elastic search, and everything that came after was also on queue.

I resolved it by killing the process.

Chidi Jude
  • 11
  • 2