Its possible to use a logback appender to send logs to kafka/rabbit-mq and then to Logstash (Elastic stack)?

Question

Context

• Microservices in the Cloud
• Components develop with Spring Boot, logback.xml
• We need Log Aggregation

Currently I can do it through TCP but I think this is not scalable enough in the AWS cloud, can I send the logs using some Queue?

Microservice > appender logback.xml > Queue > Logstash > Elasticsearch

Appender

    <appender name="STASH"
              class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>localhost:5000</destination>

        <encoder
                class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
            <providers>
                <mdc/>
                <context/>
                <logLevel/>
                <loggerName/>

                <pattern>
                    <pattern>
                        {
                        "serviceName": "clients-component",
                        "type": "business-capability"
                        }
                    </pattern>
                </pattern>

                <threadName/>
                <message/>
                <logstashMarkers/>
                <stackTrace/>
            </providers>
        </encoder>
    </appender>

logstash.conf

input { tcp { port => 5000 codec => "json" } } 

output { elasticsearch { hosts => ["localhost:9200"]} }

Answer 1

You can configure Logstash to consume from message queues without any problem.

Currently Logstash has inputs for some message queues like Kafka, RabbitMQ and Redis.

Since you are using AWS, logstash also has input for AWS SQS.

You can check all logstash inputs in the official documentation