How to dynamically provide N codecs to process fields as a VectorCodec for a record of binary fields that do not contain size bytes

Question

Considering this function in Decoder:

  final def decodeCollect[F[_], A](dec: Decoder[A], limit: Option[Int])(buffer: BitVector)(implicit cbf: Factory[A, F[A]]): Attempt[DecodeResult[F[A]]] = {

What I really need is dec: Vector[Decoder[A]], like this:

  final def decodeCollect[F[_], A](dec: Vector[Decoder[A]], limit: Option[Int])(buffer: BitVector)(implicit cbf: Factory[A, F[A]]): Attempt[DecodeResult[F[A]]] = {

to process a binary format that has fields that are not self describing. Early in the file are description records, and from these come field sizes that have to be applied later in data records. So I want to build up a list of decoders and apply it N times, where N is the number of decoders.

I could write a new function modeled on decodeCollect, but it takes an implicit Factory, so I probably would have to compile the scodec library and add it.

Is there a simpler approach using what exists in the scodec library? Either a way to deal with the factory or a different approach?

score 0 · Answer 1 · answered Jan 10 '20 at 04:06

I finally hacked a solution in the codec codebase. Now that that door is open, I'll add whatever I need until I succeed.

  final def decodeNCollect[F[_], A](dec: Vector[Decoder[A]])(buffer: BitVector)(implicit cbf: Factory[A, F[A]]): Attempt[DecodeResult[F[A]]] = {
    val bldr = cbf.newBuilder
    var remaining = buffer
    var count = 0
    val maxCount = dec.length
    var error: Option[Err] = None
    while (count < maxCount && remaining.nonEmpty) {
      dec(count).decode(remaining) match {
        case Attempt.Successful(DecodeResult(value, rest)) =>
          bldr += value
          count += 1
          remaining = rest
        case Attempt.Failure(err) =>
          error = Some(err.pushContext(count.toString))
          remaining = BitVector.empty
      }
    }
    Attempt.fromErrOption(error, DecodeResult(bldr.result, remaining))
  }

  final def encodeNSeq[A](encs: Vector[Encoder[A]])(seq: collection.immutable.Seq[A]): Attempt[BitVector] = {
    if (encs.length != seq.length)
      return Attempt.failure(Err("encodeNSeq: length of coders and items does not match"))
    val buf = new collection.mutable.ArrayBuffer[BitVector](seq.size)
    ((seq zip (0 until encs.length)): Seq[(A, Int)]) foreach { case (a, i) =>
      encs(i).encode(a) match {
        case Attempt.Successful(aa) => buf += aa
        case Attempt.Failure(err) => return Attempt.failure(err.pushContext(buf.size.toString))
      }
    }
    def merge(offset: Int, size: Int): BitVector = size match {
      case 0 => BitVector.empty
      case 1 => buf(offset)
      case n =>
        val half = size / 2
        merge(offset, half) ++ merge(offset + half, half + (if (size % 2 == 0) 0 else 1))
    }
    Attempt.successful(merge(0, buf.size))
  }


private[codecs] final class VectorNCodec[A](codecs: Vector[Codec[A]]) extends Codec[Vector[A]] {

  def sizeBound = SizeBound(0, Some(codecs.length.toLong))

  def encode(vector: Vector[A]) = Encoder.encodeNSeq(codecs)(vector)

  def decode(buffer: BitVector) =
    Decoder.decodeNCollect[Vector, A](codecs)(buffer)

  override def toString = s"vector($codecs)"

}

  def vectorOf[A](valueCodecs: Vector[Codec[A]]): Codec[Vector[A]] =
    provide(valueCodecs.length).
      flatZip { count => new VectorNCodec(valueCodecs) }.
      narrow[Vector[A]]({ case (cnt, xs) =>
        if (xs.size == cnt) Attempt.successful(xs)
        else Attempt.failure(Err(s"Insufficient number of elements: decoded ${xs.size} but should have decoded $cnt"))
      }, xs => (xs.size, xs)).
      withToString(s"vectorOf($valueCodecs)")

How to dynamically provide N codecs to process fields as a VectorCodec for a record of binary fields that do not contain size bytes

1 Answers1