Mass filtering with protobuf-net

Question

I have serialized a list of objects with protobuf-net.

Theoretically, the .bin file can contain millions of objects.

Let's assume the objects are of a class containing the following:

public string EventName;

I have to take a query and create a list containing the objects matching the query. What is the correct way to extract the matching objects from the serialized file using LINQ?

score 6 · Accepted Answer · answered Feb 14 '11 at 19:28

6

The protobuf format is a linear sequence of items; any indexing etc you way can only be applies separately. However, IEnumerable<T> is available; you might find that:

var item = Serializer.DeserializeItems<YourType>(source)
       .First(item => item.Id == id);

does the job nicely; this:

is lazily spooled; each item is yielded individually, so you don't need a glut of memory
is short-circuited; if the item is found near the start, it'll exit promptly

Or for multiple items:

var list = Serializer.DeserializeItems<YourType>(source)
    .Where(item => item.Foo == foo);

(add a ToList to te end of the above if you want to buffer the matching items in memory, or use without a ToList if you just want to parse it once in a forwards-only way)

answered Feb 14 '11 at 19:28

Marc Gravell

1,026,079
266
2,566
2,900

I think this solution is nice, but the code should be able to extract a number of objects matching the query. – Gilad Naaman Feb 14 '11 at 19:31
Thank you very much, Mark :). Again. What should I pass as argument for "tag" in DeserializeItems methods? – Gilad Naaman Feb 14 '11 at 19:42
I did. What should I pass tag in deserializeItems method? – Gilad Naaman Feb 14 '11 at 19:43
@Gilad - `Serializer.ListItemTag` (which is actually just `1`) - and the prefix-style is base-128 (but that is the default anyway) – Marc Gravell Feb 14 '11 at 19:44

score 2 · Answer 2 · answered Jun 22 '13 at 08:40

If you want to add some projection over the selected list of elements you should try a library of mine, https://github.com/Scooletz/protobuf-linq. They are available on NuGet as well. The library lowers overhead of deserialization greatly. In some cases it can drop to 50% of the original query.

score 0 · Answer 3 · answered Feb 14 '11 at 19:15

0

Unfortunately, there isn't one. In order to use LINQ, your object must implement either IQueryable<T> or IEnumerable<T>. Unless there is a LINQ provider that can provide an IQueryable<T> interface into your .bin file, you'll either have to:

Deserialize the file into memory and use LINQ-to-objects IEnumerable<T>
Write your LINQ provider that can provide an IQueryable<T> (and this is likely the only practical option if your file is HUGE) that can process the file without loading the whole thing.

answered Feb 14 '11 at 19:15

Adam Robinson

182,639
35
285
343

A good job that DeserializeItems provides an `IEnumerable` API, then – Marc Gravell Feb 14 '11 at 19:29
@Marc: It would appear so! I had assumed that deserialization would load the entire file into memory. Evidently I was wrong on that. – Adam Robinson Feb 14 '11 at 19:51
it will lazily spool them (`yield return`) from the stream - ideal for processing large data volumes – Marc Gravell Feb 14 '11 at 19:55

score 0 · Answer 4 · answered Feb 14 '11 at 19:22

0

protobuf can give you the contents of the files as a streaming IEnumerable<T>, so you can easily do that. Unfortunately I do not know how the method is called but it is easy to find in the docs.

answered Feb 14 '11 at 19:22

usr

168,620
35
240
369

@Marc how should I serialize items to use DeserializeItems? Do you have a quick example anywhere? I'm stuck trying to do just this in V2. – gjvdkamp Aug 27 '11 at 08:43

Mass filtering with protobuf-net

4 Answers4

Linked