How can I programmatically log numeric fields into Graylog?

Question

The situation

I write libraries to work against a computing server. I am logging computing times (start and stop of a job, for example). I want to log this values independently of the log framework configuration of the client application. I have achieved this by using Graylog programmatically. So I configure and initialize my logger from my library while the client remain oblivious.

This is my initialization code

import org.graylog2.log.GelfAppender;

void init() {
    final GelfAppender appender = new GelfAppender();

    appender.setName("MyServerAppender");
    appender.setGraylogHost("bigcpuserver");
    appender.setGraylogPort(2020);
    appender.setExtractStacktrace(true);
    appender.setAddExtendedInformation(true);
    appender.setAdditionalFields("{'environment': 'TEST',"
            + "'ip_address': '10.20.30.40',"
            + "'serverName': 'devPc',"
            + "'libVersion': '1.0.0',"
            + "'application': 'loggingTestApp',"
            + "'appversion': '2.0.7'}");
    appender.activateOptions();
    final org.apache.log4j.Logger myLog =
        org.apache.log4j.Logger.getLogger(MyHiddenLoggerClass.class);
    myLog.addAppender(appender);

}

So far so good. I can send logs to Graylog from anywhere in my code by means of the following line:

org.apache.log4j.Logger.getLogger(MyHiddenLoggerClass.class).info("message");

The problem

I want to add numeric fields to my logs so Graylog can run statistics on them. Like, for example, find which jobs took longest --might need algorithm optimization-- or shortest --might not need the computation server at all--.

I can add fields to the MDC in the following fashion:

org.apache.log4j.MDC.put("cpuTime", startTimeMillis-endTimeMillis);

And, from then on, all logs will include the cpuTime field with that value. I do remove it after the specific log entry has been sent to prevent carrying it over the subsequent log entries:

org.apache.log4j.MDC.remove("cpuTime");

But, to Graylog, those are strings, so it can only count them and see how many different there are.

The question

How can I tell Graylog that "cpuTime" will always be a long?

The attempts so far

I tried configuring the fields beforehand, at init() time.

To do that I tested other class hierarchies too, like me.moocar. For example, I have tried the following:

    final me.moocar.logbackgelf.GelfAppender appender =
        new me.moocar.logbackgelf.GelfAppender();

    appender.setName("MyServerAppender");
    appender.setIncludeFullMDC(true);
    appender.setGraylog2ServerHost("bigcpuserver");
    appender.setGraylog2ServerPort(2020);
    final Map<String, String> additionalFields = new HashMap<String, String>();
    final Map<String, String> fieldTypes = new HashMap<String, String>();
    additionalFields.put("cpuTime", "42");
    fieldTypes.put("cpuTime", "long");
    appender.setAdditionalFields(additionalFields);
    appender.setFieldTypes(fieldTypes);
    appender.start();

    final ch.qos.logback.classic.LoggerContext logCtx =
            (LoggerContext) LoggerFactory.getILoggerFactory();
    appender.setContext(logCtx);

    ch.qos.logback.classic.Logger logToConfigure =
            logCtx.getLogger(MyHiddenLoggerClass.class);
    logToConfigure.addAppender(appender);

So I can do the following in my test code:

    final ch.qos.logback.classic.Logger log =
             logCtx.getLogger(MyHiddenLoggerClass.class);
    MDC.put("cpuTime", totalTime);
    log.error("task finished.");

And in Graylog, the log "task finished." will be accompanied by a field named "cpuTime", and containing a string with the value of totalTime at the log moment. I just want the contained value to be a number. Anyone can help me with that?

I can change the whole to the me.moocar hierarchy if that helps at all. But so far, neither me.moocar nor org.graylog2.log have given me the result I want; and with the latter I get an easier task of attaching the appender to the logger.

update

me.moocar seems to be limited to String as type of the additional fields. So if I want numeric fields I have to choose other alternative.

Answer 1

You need to send the values according to their data types. There's a difference in

{'cpuTime':'42'}

and

{'cpuTime':42}

I don't know whether there's a transformer available in Graylog (you can handle field transformations in logstash via filters). If you want to solve the issue at logger level take a look at http://logging.paluch.biz/examples/logback.html, specifically additionalFieldTypes.

Answer 2

I've faced same issue and figured out that recent version of https://mvnrepository.com/artifact/de.siegmar/logback-gelf/4.0.0 implements the feature of sending numeric-like values as numbers to Graylog.

It's disabled by default and you can enable it by setting flag numbersAsString, for example like that in logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include
        resource="org/springframework/boot/logging/logback/base.xml" />

    <appender name="GELF"
        class="de.siegmar.logbackgelf.GelfUdpAppender">
        <graylogHost>${graylog-host}</graylogHost>
        <graylogPort>${graylog-port></graylogPort>
        <encoder class="de.siegmar.logbackgelf.GelfEncoder">
            <numbersAsString>false</numbersAsString>
        </encoder>
    </appender>
    <root level="INFO">
        <appender-ref ref="CONSOLE" />
        <appender-ref ref="GELF" />
    </root>

</configuration>

With above setup parsable values are sent to Graylog as numbers, it's handled by this piece of code in de.siegmar.logbackgelf.GelfEncoder - converts String to BigDecimal

    private Object processValue(final String value) {
        if (!numbersAsString) {
            try {
                return new BigDecimal(value);
            } catch (final NumberFormatException e) {
                return value;
            }
        }
        return value;
    }

That is later on encoded to Json as number in de.siegmar.logbackgelf.SimpleJsonEncoder

    SimpleJsonEncoder appendToJSON(final String key, final Object value) {
        if (closed) {
            throw new IllegalStateException("Encoder already closed");
        }
        if (value != null) {
            appendKey(key);
            if (value instanceof Number) {
                sb.append(value.toString());
            } else {
                sb.append(QUOTE).append(escapeString(value.toString())).append(QUOTE);
            }
        }
        return this;
    }