0

I need to implement a feature that will log data packets received from an external device to a file. Later on this file should be used for replaying this data to simulate the device. It's not possible to hold all data in memory so I need process data packet by packet.

I'm looking for a serialization mechanism that supports adding an unknown number of packets to a file and later on reload it again packet by packet. Ideally the output is human readable e.g. json but this is not a must have.

I've had a look at scala pickling but I think I would need all data in memory. Are there any better/easier ways other than manually implementing this? Would this kind of stream processing be possible using google protobuf or any related library?

user1587480
  • 1
  • 2
  • Are your packets have nested or flat structure? – om-nom-nom Sep 24 '13 at 11:22
  • The packets typically contain numbers, and lists of numbers. I'm not sure what you mean by nested. Packets can't be contained inside other packets if you thought about that. – user1587480 Sep 24 '13 at 11:25
  • I mean packets could have it's fields as all-primitive (flat), and could have some field(s) like account, which in turn will have it's own fields. I'm asking because if you have all-primitive set of fields it is insanely easy and natural to store them as csv. – om-nom-nom Sep 24 '13 at 11:30
  • Yes, it is easy but a lot of manual work because of the many different packets. – user1587480 Sep 24 '13 at 11:32

1 Answers1

0

Why not use combinators and store data in JSON files?

E.g. to parse JSON (it seems that you have a similar structure):

import scala.util.parsing.combinator._
class JSON extends JavaTokenParsers {
  def value:   Parser[Any] = obj | arr | stringLiteral | 
                             floatingPointNumber | "null" | "true" | "false"
  def obj:     Parser[Any] = "{"~repsep(member, ",")~"}"
  def arr:     Parser[Any] = "["~repsep(value, ",")~"]"
  def member:  Parser[Any] = stringLiteral~":"~value
}

Then replay by loading JSON from those files:

import java.io.FileReader
object ParseJSON extends JSON {
  def main(args: Array[String]) {
    val reader = new FileReader(args(0))
    // parseAll is overloaded: takes sequence or input reader as a second argument
    println(parseAll(value, reader))
  }
}

Examples from programming in Scala SE.

Marko Bonaci
  • 5,622
  • 2
  • 34
  • 55
  • I would have to write out JSON manually because I can't have the whole list of objects in memory. Later on I'd have to map it back to scala objects manually. I'd like to skip those steps if possible and just let a library do that for me. – user1587480 Sep 27 '13 at 10:48