0

I'm currently attempting to use the logback-logstash-encoder to write my logs to two different logstash instances. Both of these instances will be writing to the same Elasticsearch instance.

I'm struggling to find a way to load balance between the two logstash instances.

After reading both the logback documentation and the log4j2 documentation, its clear that the TcpAppender that logback-logstash uses does not support 'load-balanced' urls (i.e. url1, url2). In log4j2, I can approximate this behavior with the FailoverAppender.

Is there similar functionality in logback? Or will I need to stand up another service to load-balance for logback?

cTwospoons
  • 3
  • 3

1 Answers1

0

AFAIK there is no support for load balancing within TcpAppenders. However, you could achieve load balancing by adding a hardware/software load balancer in front of the logstash boxes. In case a logstash process/box dies, the TCP connection gets reset, and the next log event should cause to re-establish the connection again.

This approach would require adding a load balancer to your setup. You would use the load balancer's address in your TcpAppender. The balancer handles the availability of logstash for you.

It depends on what you want to achieve. Is it not losing any log events? Then perhaps the better approach would be to write the logs into files and push the files by logstash-forwarder to your logstash boxes. If you're concerned about connection timeouts and slowdown of your application, then either async-appender or using UDP could be your choice.

HTH, Mark

mp911de
  • 17,546
  • 2
  • 55
  • 95
  • Sorry for the long delay in response. This was the solution (load-balancer) we ended up going with as it allowed a fairly quick roll-out of the service without too much headache of infrastructure configuration. The main goal we wanted to achieve was high availability in our logging system and the dedicated load balancer option allowed that with no custom development efforts for logstash. – cTwospoons Jun 03 '15 at 16:44