How to parse arbitrary protobuf message in scala usind scalapb

Question

According to the documentation scalapb is able to parse arbitrary protobuf messages into json format. However, I cannot make heads or tails of their documentation and my code simply does not compile:

// apparently the Companion Object provides the parser
// and com.google.protofbuf.Any is a catch-all for protobuf messages of unknown content
val proto : protobuf.Any = com.google.protobuf.Any.parseFrom(request.contentRaw)
// according to the documentation you can convert a given parsed protobuf to json using this printer
val json = new scalapb.json4s.Printer(preservingProtoFieldNames = true)
// now I want to print the parsed message 
printer.print(proto)

but now the compiler tells me that I have parsed the wrong type?

[error]  found   : com.google.protobuf.Any
[error]  required: scalapb.GeneratedMessage
[error]     printer.print(proto)

How can I parse and print as json an arbitrary and unknown protobuf using scalapb?

You get a compile error because the code is mixing Scala and Java protocol buffer implementations. You are parsing bytes into a Java com.google.protobuf.Any and later pass it to ScalaPB which expects a Scala protobuf. The code would have compiled if you used com.google.protobuf.any.Any which is the ScalaPB implementation of Any. — thesamet, Jan 05 '23 at 17:12
However, you have incorrect assumptions on what `google.protobuf.Any` is. The documentation does not claim of "parsing arbitrary protobuf messages" using `Any`. `Any` is just a message that contains a type url and bytes. You need to be able to interpret the type url to be able to parse the bytes. Read more here: https://developers.google.com/protocol-buffers/docs/proto3#any — thesamet, Jan 05 '23 at 17:19
Yes I misunderstood the API/documentation on that type. Thank you for confirming my suspicion. — Sim, Jan 05 '23 at 17:43

score 0 · Answer 1 · answered Jan 05 '23 at 16:51

It seemed faster to write it myself, as I could not figure it out. If there is a canonical solution, I am more than happy to accept the corresponding answer. But for now:

import com.google.protobuf.{ByteString, InvalidProtocolBufferException}
import spray.json.{JsArray, JsNumber, JsObject, JsString, JsValue}
import scala.collection.mutable.{ListBuffer, Map => MMap}
import scala.jdk.CollectionConverters.{CollectionHasAsScala, MapHasAsScala}

object TestStuff {

  def createJsonObject(fieldSet : com.google.protobuf.UnknownFieldSet) : JsObject = {
    val retMap : MMap[Int,ListBuffer[JsValue]] = MMap()
    def addOrCreate(number : Int, element : JsValue) : Unit = {
      retMap.get(number) match {
        case Some(value) => value.addOne(element)
        case None => retMap.addOne(number -> ListBuffer(element))
      }
    }
    def addVarInt(number : Int, list : List[java.lang.Long]) : Unit = {
      list.foreach(elem => addOrCreate(number,JsNumber(elem)))
    }
    def addFixed32(number : Int, list : List[java.lang.Integer]) : Unit = {
      list.foreach(elem => addOrCreate(number,JsString(s"0x%08x".format(elem))))
    }
    def addFixed64(number : Int, list : List[java.lang.Long]) : Unit = {
      list.foreach(elem => addOrCreate(number,JsString(s"0x%016x".format(elem))))
    }
    def addVariableLength(number : Int, list : List[ByteString]) : Unit = {
      list.foreach {
        elem =>
          try {
            val message = com.google.protobuf.UnknownFieldSet.parseFrom(elem)
            addOrCreate(number, createJsonObject(message))
          } catch {
            case _: InvalidProtocolBufferException =>
              addOrCreate(number, JsString(com.google.protobuf.TextFormat.escapeBytes(elem)))
          }
      }
    }
    def addUnknownFieldSet(number : Int, fieldSet : com.google.protobuf.UnknownFieldSet) : Unit = {
      addOrCreate(number,createJsonObject(fieldSet))
    }
    fieldSet.asMap().asScala.foreach {
       case (number : Integer,value : com.google.protobuf.UnknownFieldSet.Field)  =>
         addVarInt(number,value.getVarintList.asScala.toList)
         addFixed32(number,value.getFixed32List.asScala.toList)
         addFixed64(number,value.getFixed64List.asScala.toList)
         addVariableLength(number,value.getLengthDelimitedList.asScala.toList)
         value.getGroupList.forEach(gle => addUnknownFieldSet(number,gle))
    }
    JsObject(
      retMap.map(elem => elem._1.toString -> JsArray(elem._2.toVector)).toMap
    )
  }

  val protoBufByteArray = getYourByteArray()
  val fieldSet = com.google.protobuf.UnknownFieldSet.parseFrom(protoBufByteArray)
  println(createJsonObject(fieldSet).prettyPrint)
}

How to parse arbitrary protobuf message in scala usind scalapb

1 Answers1