Integrating logstash (elk) with mulesoft

Question

Is there a way we could feed the mule cloudhub logs to logstash?

I am able to configure agents as in this link

https://docs.mulesoft.com/cloudhub/sending-data-from-arm-to-external-monitoring-software

But I am unable to configure logstash.

Answer 1

You could use log4j socket appender, if cloudhub allows connecting to outside server. You could configure logstash to listen to tcp for logs, you could refer to this, https://www.elastic.co/guide/en/logstash/current/plugins-inputs-tcp.html. Here is an example of tcp input for logstash(https://gist.github.com/jgoodall/6323951), make sure you set the correct format for mule because there would be a problem with multiline data. Also configure your app, for older version of mule, you could add log4j.xml

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
  <appender name="console" class="org.apache.log4j.ConsoleAppender">
    <param name="Target" value="System.out" />
    <layout class="org.apache.log4j.PatternLayout">
        <param name="ConversionPattern" value="%d{dd-MM-yyyy HH:mm:ss} %-5p [%t] [%c] - %m%n" />
    </layout>
    <filter class="org.apache.log4j.varia.LevelRangeFilter">
        <param name="LevelMin" value="ERROR" />
        <param name="LevelMax" value="FATAL" />
        <param name="AcceptOnMatch" value="true" />
    </filter>
</appender>
<appender name="socket" class="org.apache.log4j.net.SocketAppender">
    <param name="Threshold" value="INFO" />
    <param name="application" value="AppName" />
    <param name="Port" value="4560" />
    <param name="RemoteHost" value="remotehost.com" />
    <param name="ReconnectionDelay" value="60000" />
    <layout class="org.apache.log4j.PatternLayout">
        <param name="ConversionPattern" value="%d{dd-MM-yyyy HH:mm:ss} %-5p [%t] [%c] - %m%n" />
    </layout>

</appender>

 <appender name="asyncConsole" class="org.apache.log4j.AsyncAppender">
    <param name="BufferSize" value="200"/>
    <appender-ref ref="console"/>
</appender>

<appender name="asyncFile" class="org.apache.log4j.AsyncAppender">
    <param name="BufferSize" value="200"/>
    <appender-ref ref="file"/>
</appender>

<appender name="asyncSocket" class="org.apache.log4j.AsyncAppender">
    <param name="BufferSize" value="200"/>
    <appender-ref ref="socket"/>
</appender>

<logger name="Tracking"  additivity="false">
    <level value="warn" />
    <appender-ref ref="asyncConsole" />
    <appender-ref ref="asyncFile" />
    <appender-ref ref="asyncSocket" />
</logger>

<logger name="com.mulesoft.module.datamapper"  additivity="false">
    <level value="warn" />
    <appender-ref ref="asyncConsole" />
    <appender-ref ref="asyncFile" />
    <appender-ref ref="asyncSocket" />
</logger>

<logger name="org.mule.api.processor.LoggerMessageProcessor" additivity="false">
    <level value="info" />
    <appender-ref ref="asyncConsole" />
    <appender-ref ref="asyncFile" />
    <appender-ref ref="asyncSocket" />
</logger>
<root>
    <priority value="info" />
    <appender-ref ref="asyncConsole" />
    <appender-ref ref="asyncFile" />
    <appender-ref ref="asyncSocket" />
</root>

Notice the socket appender, from there you could configure the host and port of the logstash instance. For newer version of mule, there is a log4j2.xml, the configuration for the socket appender is similar.