API Logging in Json format

Question

I would like to log API request/response as json format.

The expected LogEntry is something like

{ 
 "timestamp" : "...",
 "level" : "DEBUG",
 "headers" : [
    "header1" : "value1",
    "header2" : "value2",
    "header3" : "value3"
 ],
 "requestPayload" : "<Request Json>"   // prefereablly as sub-document, worst case string is fine.   
 "labels" : [        //key fields which can be used for searching the logentry
     "searchField1" : "....",
     "searchField2" : "....",
     "searchField3" : "...."
  ]
}

My Question is :

Using Logback, How to log nested fields (e.g. headers, labels, requestPaylod in above example), as json sub-document. I tried MDC, but its limited to Map of 'String, String' only, and considers all fields after first level as String.

I hate to write my custom logger for this, and would like to use the goodness of proven logging frameworks(logback/log4j) to control the logging level, time stamping log event, etc.

score 4 · Answer 1 · answered Apr 20 '18 at 10:33

4

For request/response logging try Logbook and logstash-logback-encoder. You'll be able to use Markers to add structured contents.

For adding 'your own' fields in a JSON structure, I've written a code generator you might find interesting, it adds builder support: json-log-domain

answered Apr 20 '18 at 10:33

ThomasRS

8,215
5
33
48

1

Hey! Markers are exactly what I was looking for! – psv Jan 21 '21 at 18:26

glytching · Answer 2 · 2017-08-27T17:07:41.433

You can use logback-contrib's JsonLayout inside any Logback appender. For example:

<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
        <jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
            <prettyPrint>false</prettyPrint>
        </jsonFormatter>
        <timestampFormat>yyyy-MM-dd' 'HH:mm:ss.SSS</timestampFormat>
        <appendLineSeparator>true</appendLineSeparator>
    </layout>
</appender>

Given log statements like this ...

MDC.put("header1", "headerValue1");
logger.info("hello!");
logger.info("good bye!");

... the use of JsonLayout would result in Logback writing this:

{"timestamp":"2017-08-15 09:06:41.813","level":"INFO","thread":"main","mdc":{"header1":"headerValue1"},"logger":"com.stackoverflow.logback.LogbackTest","message":"hello!","context":"default"}
{"timestamp":"2017-08-15 09:06:41.887","level":"INFO","thread":"main","mdc":{"header1":"headerValue1"},"logger":"com.stackoverflow.logback.LogbackTest","message":"good bye!","context":"default"}

I think this ticks the boxes of writing your log events as JSON documents whilst still retaining Logback's behaviour such as "control the logging level, time stamping log event, etc". There is some built-in support for changing the JSON format (e.g. you can include/exclude context, logger name etc) but the JsonLayout class provides an extension point which would allow you to change the attribute names in the resulting JSON by extending JsonLayout and overriding toJsonMap().

Edit 1: addressing your reply ("my question was more on how to achieve json subdocument/nesting as mentioned. MDC is limited to map of only") ... you could serialise your complex MDC values to JSON and add the serialised JSON representations to MDC. For example:

    Map<String, Object> complexMdcValue = new HashMap<>();
    Map<String, Object> childMdcValue = new HashMap<>();
    childMdcValue.put("name", "Joe");
    childMdcValue.put("type", "Martian");
    complexMdcValue.put("child", childMdcValue);
    complexMdcValue.put("category", "etc");

    MDC.put("complexNestedValue", objectMapper.writeValueAsString(complexMdcValue));
    logger.info("hello!");

Would produce this output (in which the MVC logged key "complexNestedValue" is JSON containing a sub document):

{"timestamp":"2017-08-27 18:03:46.706","level":"INFO","thread":"main","mdc":{"complexNestedValue":"{\"category\":\"etc\",\"child\":{\"name\":\"Joe\",\"type\":\"Martian\"}}"},"logger":"com.stackoverflow.logback.LogbackTest","message":"hello!","context":"default"}

Well, I could achieve this earlier, but my question was more on how to achieve json subdocument/nesting as mentioned. MDC is limited to map of only — kiran, Aug 27 '17 at 03:37
I've updated the answer to suggest a way to write complex MDC values as Strings. — glytching, Aug 27 '17 at 17:08
Well, this does print the complex json but the value is just a string and needs to be parsed again in json explicitly. Also it dilutes the purpose of using MDC this way, MDC keys are helpful in controlling logs patterns declaratively, which is not possible with this approach. — kiran, Aug 27 '17 at 18:38
Yeah, that's all true but since MDC is typed to String your options are limited. Either 1/ represent this complex state as a String and accept the need for a subsequent deserialisation call or 2/ provide your own implementation of something like MDC or 3/ perhaps reconsider whether MDC is the right 'fit' for this use case. — glytching, Aug 27 '17 at 18:43
Escaped json isn't much use... especially if you're intending to get something searchable — Edd, Jul 13 '21 at 11:33

API Logging in Json format

2 Answers2