Scala adding an element to JsValue before convertT[class]

Question

I have a Kafka consumed record that will be parsed as JsValue with spray.json in scala, but I also have some data in the record's header, and I want to do:

Consume record with Alpakka Kafka library (done)
parse as json of type JsValue: kafkaRecord.record.value().parseJson (also done)
Append the record's header to that JsValue (HERE IS THE MAIN CONCERN)

Header = kafkaRecord.record.headers()

Appending should include key[String]: value(header)
convert to pre-defined case class using [JsValue].convertTo[<case class>]

Here is the consumed record for example:

{"id": 23, "features": "features_23"}

and want to append to it the Header to be as:

{"id": 23, "features": "features_23", "header_data":"Header_23"}

Then convert to case class:

case class recordOfKafka(id: Int, features: String, header_data: String)

score 1 · Accepted Answer · answered Jul 21 '22 at 08:54

1

Assuming some functions that get the string value of the record and header:

@ def getKafkaRecord(): String = """
  {"id":1,"features":"feature_23"}
  """ 
defined function getKafkaRecord

@ def getKafkaHeader(): String = """
  {"header_data":"header_23"}
  """ 
defined function getKafkaHeader

And the model with formatter:

@ import spray.json._
@ import DefaultJsonProtocol._

@ case class SomeModel(id: Int, features: String, header_data: String) 
defined class SomeModel

@ implicit val someModelFormatter = jsonFormat3(SomeModel) 
someModelFormatter: RootJsonFormat[SomeModel] = spray.json.ProductFormatsInstances$$anon$3@6c1e40d9

You can parse record value and header in separate json objects, and then combine their fields together:

@ // parse as json object, convert to map
@ JsObject(
    getKafkaRecord.parseJson.asJsObject.fields ++
      getKafkaHeader.parseJson.asJsObject.fields
  ).convertTo[SomeModel] 
res15: SomeModel = SomeModel(
  id = 1,
  features = "feature_23",
  header_data = "header_23"
)

One thing to take care about is that, if you want to use .asJsObject you must be sure that the records are json objects, like if you receive:

[{"key":"value"},{"key":"value_2"}]

It'll throw exceptions since this is an json array and cannot be converted to json object! You can also use try or pattern matching for conversion to JsObject to be safer.

answered Jul 21 '22 at 08:54

AminMal

3,070
2
6
15

Following to your last note, I am sure that the record's value will be a JSON object, but the record's header I am sure that it will be not. I believe this will cause an exception. For now, I solved it by having two case classes, one case class for the attributes in the record's value, and the other case class is for the first case class + the record's header. So, the second case class will hold all attributes but in a nested json. What do you think of this work-around? – Omar AlSaghier Jul 21 '22 at 13:39
What I was thinking of, is having all `Header` as a value of custom key called `"metadata"` – Omar AlSaghier Jul 21 '22 at 13:43
That’s also reasonable, actually that was my first solution but didn’t want to complicate the answer. Can you also provide some real sample of the headers? Maybe there was a better approach to your problem. – AminMal Jul 21 '22 at 13:50
Honestly, I also thought that you will provide that solution when I first read the first part of the answer. Also, regarding the header I am still not sure about that since my app part is only a consumer, that's why I saved it as a whole as a string in the case class for now. – Omar AlSaghier Jul 21 '22 at 14:04
You can also statically put the header with the header key as a tuple, and concatenate it with the actual record value, like: ‘getRecord.parseJson.asJsObject.fields + (“header_data” -> getHeader)’ (you might also need to explicitly use JsString constructor for the header value, I couldn’t check since I don’t have access to my computer rn) – AminMal Jul 21 '22 at 18:56
That's right. I will also accept this answer as it satifies the questions generally, but my case is that I am not sure about the header if it is Json. Thank you @AminMal – Omar AlSaghier Jul 21 '22 at 19:12

score 0 · Answer 2 · answered Jul 25 '22 at 06:37

The above answer is true, but in case someone is not 100% sure about the JSON format, here is another solution I figured out:

I implemented two case classes, the 1st case class to parse Kafka record without the header:

case class KafkaRecordWithoutHeader(KafkaRecordId: String,
                             KafkaRecordIndex: Long,
                             KafkaRecordUserId: String,
                             statusCode: Int,
                             KafkaRecordData: String)

And another case class that holds the 1st case class, and the header:

case class KafkaRecordWithHeader(recordValue: KafkaRecordWithoutHeader, 
                                 KafkaHeader: String)

Then, parsing Kafka record inside the consumer as:

kafkaRecordValue = kafkaRecord.record.value().parseJson.convertTo[KafkaRecordWithoutHeader]

And parse the KafkaRecordWithoutHeader class with the KafkaHeader in the main case class:

kafkaRecordWithHeader = KafkaRecordWithHeader(kafkaRecordValue, kafkaRecord.record.headers().toString)

Scala adding an element to JsValue before convertT[class]

2 Answers2