How to encode a raw, unescaped JSON string?

Question

I have a Kotlin server that acts as a gateway, handling communication between a client and a number of backing services over a REST APIs, using JSON. My server uses Kotlinx Serialization for serialization.

Usually I need to parse and adapt the responses from the backing services, but occasionally I just want to return the raw JSON content as a response.

For example:

import kotlinx.serialization.json.*

fun main() {
  // I get some JSON from a backing service
  val backingServiceResponse = """
    {"some":"json",id:123,content:[]}
  """.trimIndent()

  // I create a response object, that I will return to the client
  val packet = ExampleClientResponse("name", backingServiceResponse)

  val encodedPacket = Json.encodeToString(packet)

  println(encodedPacket)

  // I expect that the JSON is encoded without quotes
  require("""{"name":"name","content":{"some":"json",id:123,content:[]}}""" == encodedPacket)
}

@Serializable
data class ExampleClientResponse(
  val name: String,
  val content: String, // I want this to be encoded as unescaped JSON
)

However, the value of content is surrounded by quotes, and is escaped

{
  "name":"name",
  "content":"{\"some\":\"json\",id:123,content:[]}"
}

What I want is for the content property to be literally encoded:

{
  "name":"name",
  "content":{
    "some":"json",
    "id":123,
    "content":[]
  }
}

I am using Kotlin 1.8.0 and Kotlinx Serialization 1.4.1.

aSemy · Accepted Answer · 2023-02-04T09:44:53.107

Encoding raw JSON is possible in Kotlinx Serialization 1.5.0-RC, which was released on 26th Jan 2023, and is experimental. It is not possible in earlier versions.

Create a custom serializer

First, create a custom serializer, RawJsonStringSerializer, that will encode/decode strings.

Encoding needs to use the new JsonUnquotedLiteral function to encode the content, if we're encoding the string as JSON

Since the value being decoded might be a JSON object, array, or primitive, it must use JsonDecoder, which has the decodeJsonElement() function. This will dynamically decode whatever JSON data is present to a JsonElement, which can be simply be converted to a JSON string using toString().

import kotlinx.serialization.*
import kotlinx.serialization.descriptors.*
import kotlinx.serialization.encoding.*
import kotlinx.serialization.json.*

private object RawJsonStringSerializer : KSerializer<String> {

    override val descriptor = PrimitiveSerialDescriptor("my.project.RawJsonString", PrimitiveKind.STRING)

    /**
     * Encodes [value] using [JsonUnquotedLiteral], if [encoder] is a [JsonEncoder],
     * or with [Encoder.encodeString] otherwise.
     */
    @OptIn(ExperimentalSerializationApi::class)
    override fun serialize(encoder: Encoder, value: String) = when (encoder) {
        is JsonEncoder -> encoder.encodeJsonElement(JsonUnquotedLiteral(value))
        else -> encoder.encodeString(value)
    }

    /**
     * If [decoder] is a [JsonDecoder], decodes a [kotlinx.serialization.json.JsonElement] (which could be an object,
     * array, or primitive) as a string.
     *
     * Otherwise, decode a string using [Decoder.decodeString].
     */
    override fun deserialize(decoder: Decoder): String = when (decoder) {
        is JsonDecoder -> decoder.decodeJsonElement().toString()
        else -> decoder.decodeString()
    }
}

Apply the custom serializer

Now in your class, you can annotated content with @Serializable(with = ...) to use the new serializer.

import kotlinx.serialization.*

@Serializable
data class ExampleClientResponse(
  val name: String,
  @Serializable(with = RawJsonStringSerializer::class)
  val content: String,
)

Result

Nothing has changed in the main method - Kotlinx Serialization will automatically encode content literally, so it will now succeed.

import kotlinx.serialization.*
import kotlinx.serialization.descriptors.*
import kotlinx.serialization.encoding.*
import kotlinx.serialization.json.*

fun main() {
  val backingServiceResponse = """
    {"some":"json",id:123,content:[]}
  """.trimIndent()

  val packet = ExampleClientResponse("name", backingServiceResponse)

  val encodedPacket = Json.encodeToString(packet)

  println(encodedPacket)
  require("""{"name":"name","content":{"some":"json",id:123,content:[]}}""" == encodedPacket)
}

Reducing duplication

Using @Serializable(with = ...) is simple when it's a on-off usage, but what if you have lots of properties that you want to encode as literal JSON?

Typealias serialization

When a fix is released in Kotlin 1.8.20 this will be possible with a one-liner

// awaiting fix https://github.com/Kotlin/kotlinx.serialization/issues/2083
typealias RawJsonString = @Serializable(with = RawJsonStringSerializer::class) String


@Serializable
data class ExampleClientResponse(
  val name: String,
  val content: RawJsonString, // will be encoded literally, without escaping 
)

Raw encode a value class

Until Kotlinx Serialization can handle encoding typealias-primitives, you can use an inline value class, which we tell Kotlinx Serialization to encode using RawJsonStringSerializer.

@JvmInline
@Serializable(with = RawJsonStringSerializer::class)
value class RawJsonString(val content:  String) : CharSequence by content

Now now annotation is needed in the data class:

@Serializable
data class ExampleClientResponse(
  val name: String,
  val content: RawJsonString, // will be encoded as a literal JSON string
)

RawJsonStringSerializer needs to be updated to wrap/unwrap the value class

@OptIn(ExperimentalSerializationApi::class)
private object RawJsonStringSerializer : KSerializer<RawJsonString> {

  override val descriptor = PrimitiveSerialDescriptor("my.project.RawJsonString", PrimitiveKind.STRING)

  override fun deserialize(decoder: Decoder): RawJsonString = RawJsonString(decoder.decodeString())

  override fun serialize(encoder: Encoder, value: RawJsonString) = when (encoder) {
    is JsonEncoder -> encoder.encodeJsonElement(JsonUnquotedLiteral(value.content))
    else           -> encoder.encodeString(value.content)
  }
}

The tradeoff is that it's a little clunky to convert to/from the new value class.

val backingServiceResponse = """
  {"some":"json",id:123,content:[]}
""".trimIndent()

// need to wrap backingServiceResponse in the RawJsonString value class
val packet = ExampleClientResponse("name", RawJsonString(backingServiceResponse))

This answer was written using

Kotlin 1.8
Kotlinx Serialization 1.5.0-RC.

that's not possible in current projects that does not use this library — Ahmed Adel Ismail, Aug 29 '23 at 14:28