0

We have a use case of data streaming from the main transactional system to other downstream such as data analytics and machine learning team.

One of the requirements are to ensure data governance that data source can control who can read which column, and potentially lifecycle of a data to ensure data siting in another domain gets purged should the source data removed it, such as if a user deletes the account, we need to make sure the data in all downstream gets removed.

While we are considering Thrift, Avro and ProtoBuf, what are the common frameworks that we can use for such data governance? Do any of these protocol supports metadata for such data governance around data authorization, lifecycle?

RichardKang
  • 497
  • 5
  • 16

1 Answers1

0

Let me get this straight:

protobuf is not a security device; to someone with the right tools it is just as readable as xml or json, with the slight issue that it can be uncertain how to interpret some values;

It's not of a much difference than JSON nor XML. It is just an interface language. Sure, it has encoding, it is a bit different and a lot more customizable, but it does in no way confront security. It is up to you to secure the channel between sender and receiver.

Stf Kolev
  • 640
  • 1
  • 8
  • 21
  • 1
    Thanks and yes I agree that AVRO or ProtoBuf are just encoding format. Reason of asking is, we have a use case that the application team will define the data over wire and we would like to start with right foot, thus exploring what kind of encoding allows the application team to embed the ACL metadata within the data schema. – RichardKang Jul 09 '20 at 10:19
  • 1
    There is actually no wrong turn here, because in AVRO you have [message framing](https://avro.apache.org/docs/current/spec.html#Message+Framing), to where as in protobuf you have [custom options](https://developers.google.com/protocol-buffers/docs/proto#customoptions). It is up to you and your team to figure out what suits your style and what is going to be easier for all of you. Correct me if I misunderstood something – Stf Kolev Jul 09 '20 at 10:35