Are there any databases, in either the SQL or NoSQL worlds, that support protocol buffers? (Support could mean various things, but hopefully would include being able to index on fields in protocol buffers.)
Asked
Active
Viewed 1.8k times
37
-
15 mins spent googling didn't turn up any positive results. Hadoop HBase uses protocol buffers as part of the service comms, and has a Thrift client API, but nothing at the data storage layer. Hadoop Avro is in the same space as PBs, maybe we'll see integration with HBase in future? – mdma May 09 '10 at 19:05
-
Why not just use a relational database (put each value in its own column rather than all values in one column)? – Brendan Long May 30 '10 at 22:15
-
@Brendan: Of course, that's one way to go. Mostly I was surprised that none of the noSQL databases had an out-of-the-box solution, given that some, like MongoDB, store binary blobs that are similar. – Rob Lachlan May 31 '10 at 01:29
-
1Did you arrive at this requirement by thinking of versioning your data model instead of deleting it? I'm thinking of doing that. If not, is it because HBase rows can have an arbitrary number of columns in which case Protocol Buffers are an excellent fit. I'm planning on doing this programmatically though. – Ravindranath Akila Dec 19 '12 at 02:17
-
@RavindranathAkila No I wasn't thinking in terms of versioning the data model, but that's an excellent point. It's more that I was looking for a portable data interchange format, and wouldn't be nice if the database itself supported that. – Rob Lachlan Dec 19 '12 at 05:16
-
3Since it's proprietary it doesn't really help you, but Google's [F1 database](http://research.google.com/pubs/pub41344.html) supports Protocol Buffers like you envision. I'm not sure if any public databases have implemented anything similar. – dimo414 Oct 21 '14 at 05:33
5 Answers
8
ProfaneDB: http://profanedb.gitlab.io
"ProfaneDB is a gRPC interface between Protocol Buffers and RocksDB"
P.S: Though the question was asked 10 years ago I still find it very relevant, specially with the ascension of gRPC. Hope that can be of help.
Vitor Matos
- 81
- 1
- 1
5
I am not aware of any (not that they don't exist, I just don't know about them), but perhaps it is worth discussing an alternate strategy and the pros and cons.
Typically you would want to store a serialized version of your protocol buffer, indexed by some particular key (a unique identifier, perhaps). You could then build secondary indexes for other interesting fields that point to that unique identifier. The idea is that you would want explicit indexes like this to scale beyond what a system that provides arbitrary indexes. This obviously opens up a world of new problems (such as stale arbitrary indexes).
allenporter
- 361
- 1
- 4
-
2Why would anyone want keys and other entites packed into protobuf instead any language can just work with primitive types for database indexing etc. it works great of it's purpose but it doesn't mean that it should be used everywhere where it doesn't make any sense, – mamu Jun 16 '10 at 03:49
-
If you want to use your database across many machines you'll likely need to handle the secondary indexes yourself anyway instead of relying on some database software to do it fore you. – allenporter Jul 05 '10 at 02:36
-
GCP BigTable supports indexing on a key/value basis which can be used in case of protobuf, where you'd index on a particular row key(unique identifier)(if this is the kind of solution you are looking for) – Anamika Modi May 27 '20 at 06:50
2
Here is a project I found: https://github.com/google/mysql-protobuf
It's sql that speaks protobuf, looks promising but last commit was in mid March 2016.
Azr
- 1,083
- 1
- 13
- 26
2
Cloud Spanner would provide native support for protobufs.
zavidovych
- 4,718
- 2
- 24
- 22
-
2Where are you seeing that? Google's internal Spanner supports protobuffers, but Cloud Spanner team says they don't have any current plans to support protobuffers natively. – normmcgarry Apr 06 '21 at 12:56