Ensuring Data completeness and validity on third party storage

Question

I'm dealing with untrusted external storage and need to ensure the storage provider does not withhold any records in a query.

Example:

I have two trusted entities TA and TB, those entities should be able to alter the data that is stored in the cloud/untrusted storage, but nobody else. So my solution I equip TA and TB with Public-Keys and i have a data structure that can be compared to a table with versions say

 Ver | Data | Signature       | Signee
  4  |  ... | (AAAAAAAAA)_TA  | TA
  3  |  ... | (ZZZZZZZZZ)_TB  | TB
  2  |  ... | (YYYYYYYYY)_TA  | TA
  1  |  ... | (XXXXXXXXX)_TA  | TA

So when I retrieve such a table from the storage provider, I can easily verify the signatures and check whether the signature is correct, whether the signee was allowed to change the table or not.

However, I would also like to check for record completeness. Say TA uploads version 4, but TB is only aware of all records up to Version 3. Now the storage provider may withhold Version 4 completely when TB queries it.

As there is no direct sidechannel between TA and TB, there is no way to exchange the current version. Is there a way to circumvent this?

I was thinking of periodically inserting dummy records to at least have some time certainty. However, this approach lacks scalability and would result in a lot of storage and signing overhead. What is the actual system property i am looking for (it is hard to find research for something you do not know the name of)?

Answer 1

If the problem was about record integrity, I would recommend to use MAC (Message Authentication Code). In this case you should use symmetric key cryptography, and it is much more efficient than cryptographic signatures (asymmetric).

I thought about two directions:

1. You can reduce your problem to record integrity. If you have specific important data that you want to verify, you can just create specific table in the storage that saves when each of the trusted entities updated each of your important variables. Important data can be something like: number of records (total, new, according to some category), latest version, etc. When someone does a change that affects important data, he updates the relevant record in the spacial table, and sign on the record - and the record contains also the date of the last update. Now, to prevent the untrusted storage from returning old records or such things, each of the trusted parties must update the records at least once every some time period (for example - one day)- the records might not changed (only the date in the record), and the sign will change (now the sign is on later date). When entity reads information, he can verify that the data is authentic and was true for (in the example) at most one day before. If the date of the returned record is older than one day, than it should not be considered. You can change the frequency of the periodic updates according to your needs. For this option you can use MAC instead of signatures as well (to improve efficiency).

2. You can try another strategy: You cannot validate the completeness of every update, but you can try to catch the untrusted storage if it tries to lie! You can do it by scheduling updates and query for them.

Last thing - You wrote:

As there is no direct sidechannel between TA and TB, there is no way to exchange the current version. Is there a way to circumvent this?

I believe you meant communication channel. Side-channel is something else. If it was me, I would just create such communication channel to solve the problem. And by the way, similarly to the first option I describe, you can create such a channel:

Create a table of messages from the different entities where they announce about the important changes they did and sign about them with the date of the change:

Entity | Changes | Date | Signature (or MAC)

Like in the first option every entity must add such a message once every some predefined time period - and you have trusted communication channel between the entities.

Answer 2

This problem is not fully solvable without dummy records:

Let's call the state when the current version is version 3 "state 3", and the state when the current version is version 4 "state 4". No matter how you sign these states - as long as the attacker is telling you "state 3 is the current one" (showing you the entire database as it was during state 3), you can't know if this is true or if state 4 exists in the meantime.

Thus, you will have to periodically sign "no change" updates. You won't be able to avoid the signing overhead, but you don't have to store all of these. You make a separate "lastupdate" table:

 Signer | Last | Timestamp | Signature
  TA    |  4   | 2013-1... | (TA;4;2013-1...)_TA
  TB    |  3   | 2013-1... | (TB;3;2013-1...)_TB

meaning "Signer TA confirms that as of 2013-1..., the last version sent by me was 4". If the storage provider cannot show you a current confirmation from all signers that they didn't issue a newer version, you have to assume that he is hiding something (or something broke - either way, the data is not up to date). Any new signed statement replaces the older ones from that signer, because they are irrelevant now.

Now, if you don't have just one versioned "thing", but a large number of them, you don't necessarily have to have one such dummy log per "thing". For example, you could calculate a hash (or hash tree) over all the most current lines by your signer (e.g. "Thing A, Version 3. Thing B, Version 7. Thing C, Version 2.") and then just put the hash or the root of the hash tree in the lastupdate table.

If you really want to avoid additional signatures, and some things get updated all the time, you could include the hash and timestamp in the signatures of the version records you sign - the most current signed record would then be sufficient proof for freshness, and if it were too old, you could still use the "lastupdate" table. This is not worth the complexity, IMHO.