How to store protobuf object in memcached?

Question

Following is the proto file I am using to create a GRPC system which gets data from the database/memcached.

message CMSContent
{
    repeated ArticleSummary Articles = 1;
    uint32 RecordCount = 2;
}

service Article
{
    rpc ListByCategory(ArticleByCatURI) returns (CMSContent){}
}

message ArticleByCatURI
{
    uint32 ApplicationId = 1;        
}

message ArticleSummary
{
    uint32 ArticleId = 1;
    string ArticleName   = 2;
}

I created the corresponding C# classes as detailed on GRPC Getting Started. I am using these classes on Server to store the result of a DB Query and trying to save these objects to memcached. Unfortunately the protobuf object is not getting added to memcached. Following is how I am adding it to memcached and the value of isKeyFirstTimeCreated is false when I try to add it.

 isKeyFirstTimeCreated = mc.Store(StoreMode.Add, key, t, DateTime.Now.Add(cacheDuration));

As per memcached requirement, all the objects which must be stored in memcached need to be Serializable. But the autogenerated protobuf classes are not serializable (at least to binary formatter which c# uses). Now since the classes are partial implementations, I made them serializable but that did not help as the RepeatedField is not Serializable.

IMO, there should be a better way or some mechanism which will allow me to save the protobuf objects directly into the memcached but unfortunately Googling didn't help me much in that regard.

I can achieve this in two ways

by copying data from protobuf object to a serializable object and then save it into memcached for the future reference
copy the data into a serilizable object and use that object to populate the protobuf object.

But these two approached need extra work. Is there a better way to achieve this?

[PS: I am using proto3 hence it supports c#]

score 1 · Accepted Answer · edited Feb 25 '16 at 11:48

1

Almost all cache stores support byte[]. Tools like protobuf lend themself directly to being stored in a byte[], so the most common approach here with binary serializers is essentially:

byte[] raw;
using(var ms = new MemoryStream(ms)) {
    YourChosenSerializer.Serialize(ms, obj);
    raw = ms.ToArray();
}
// now store raw

If your storage tool doesn't support byte[], then string is usually a good substitute, but binary serializers should be stores using something like base-64 encoding (via Convert.ToBase64String and Convert.FromBase64String).

As for the memcached library (or similar) supporting this directly: I would advise against this for separation of concerns reasons. However, extension methods usually make it very easy to provide this in a few lines in helper methods in your own code.

edited Feb 25 '16 at 11:48

Loofer

6,841
9
61
102

answered Feb 15 '16 at 21:14

Marc Gravell

1,026,079
266
2,566
2,900

Yes this does work for me and it's much better than the two approaches I had earlier. But I was wondering if we can do something better than this too. How are you achieving the same result in _protobuf-net_? This approach will still need me to convert my Object into a byte[] array and then store the array. I was thinking that the protobuf classes being serializable in protobuf format, there might be a way to directly store those objects than converting them to byte arrays by tweaking some of the properties or by setting the transcoder as you do with protobuf-net. – GawdePrasad Feb 16 '16 at 12:13

How to store protobuf object in memcached?

1 Answers1