How to store dataset with protobuf

Question

I need to store flexible data sets compactly and with ability to streaming. Data set contain tabular data, each rows contain same types.

Whats better: - write one big proto-message with repeated proto-messages? - multiple proto messages writes directly to steram (with some additional info, about proto format inside)? - or some better way exists?

Is protobuf support streaming?

Answer 1

As you've probably noticed, protobuf is mostly a framework for serializing/deserializing data into/from a binary form. If you're looking to transfer that data, you can of course send it in a streaming way - it's just bytes after all.

You could write your own network protocol, but probably what you're looking for is something like gRPC (Link is to the C# tutorial - deduced from your tags). gRPC allows for streaming - you can learn more about it here. Quick quote:

Server streaming RPC A server-streaming RPC is similar to our simple example, except the server sends back a stream of responses after getting the client’s request message. After sending back all its responses, the server’s status details (status code and optional status message) and optional trailing metadata are sent back to complete on the server side. The client completes once it has all the server’s responses.

Client streaming RPC A client-streaming RPC is also similar to our simple example, except the client sends a stream of requests to the server instead of a single request. The server sends back a single response, typically but not necessarily after it has received all the client’s requests, along with its status details and optional trailing metadata.

Also, for a quick example, visit this page. An example from gRPC's website:

public override async Task<RouteSummary> RecordRoute(Grpc.Core.IAsyncStreamReader<Point> requestStream,
    Grpc.Core.ServerCallContext context)
{
    int pointCount = 0;
    int featureCount = 0;
    int distance = 0;
    Point previous = null;
    var stopwatch = new Stopwatch();
    stopwatch.Start();

    while (await requestStream.MoveNext())
    {
        var point = requestStream.Current;
        pointCount++;
        if (CheckFeature(point).Exists())
        {
            featureCount++;
        }
        if (previous != null)
        {
            distance += (int) previous.GetDistance(point);
        }
        previous = point;
    }

    stopwatch.Stop();

    return new RouteSummary
    {
        PointCount = pointCount,
        FeatureCount = featureCount,
        Distance = distance,
        ElapsedTime = (int)(stopwatch.ElapsedMilliseconds / 1000)
    };
}

EDIT

Example proto file with gRPC definitions in it.

syntax = "proto3";

message DataRow {
    // Your data row here
}

message SendDataResponse {
}

service DataAcceptor {
  // Send multiple rows in a streaming scenario
  rpc SendData (stream DataRow) returns (SendDataResponse) {}
}