0

Hi we have created a Syslog Drain from pcf to logstash but sometimes we are getting 2018-07-19T15:09:53.524+05:30 [LGR/] [ERR] Syslog Drain: Error when writing. Backing off for 4ms.

this error. What is this and why?

user2835563
  • 99
  • 2
  • 13

1 Answers1

1

I suspect that it's a communication problem with the logging system in your Cloud Foundry platform as it's trying to talk with your LogStash. The message doesn't give you an exact error though. To find that, you would need to be a platform operator and look at the Loggregator logs to see why it's failing. If you're not the CF platform operator, reach out to your operator for assistance.

When you see errors like this I would suggest checking for two things:

  1. How often do you see this message?
  2. How large does the number in "Backing off for XXms." get?

When an error occurs sending logs the platform will back off, but as errors continue to occur the backoff timeout will get larger. If you see a large value in the backoff timeout, that means you have a prolonged problem. This could be something like you've configured the log drain incorrectly, your LogStash server is down or the network to it is down. If you see the errors frequently, but the number stays low, it means it's only intermittently failing (some logs go OK, some don't) which could point to a flaky network connection, one that's up/down a lot.

Daniel Mikusa
  • 13,716
  • 1
  • 22
  • 28
  • Hey thanks for that detailed answer looks like every time i update my grok and restart my logstash is causing this issue. – user2835563 Jul 19 '18 at 18:22
  • That would make sense. If you're restarting the process, it would disappear, the log drain would disconnect. When the process is back up, it should automatically connect and start sending logs again. – Daniel Mikusa Jul 19 '18 at 20:08