I want to create a tool to delete all of the data older than some days in rocksdb or leveldb, but i don't know how to start it.
Asked
Active
Viewed 2,272 times
5
-
so i assumed that you want data to have ```ttl``` ? – sel-fish Sep 12 '16 at 14:25
-
What is the layout of the data you are storing in the database? – amirouche Nov 08 '16 at 19:39
-
What languages do you use? – amirouche Dec 25 '16 at 00:40
-
I use golang if you mean programming lang – Michael Feb 18 '17 at 02:13
-
1@sel-fish yes, i think ttl is what i wanted – Michael Feb 18 '17 at 02:14
-
@Michael add timestamp to leveldb/rocksdb record, and modify compact logic to drop expired record, which you can reference https://github.com/lotair/tair/tree/develop/src/storage/ldb/leveldb – sel-fish Feb 18 '17 at 08:26
1 Answers
1
RocksDB has a specific database, called Date_tiered DB. You can specify the ttl with which u want to open the database. It basically shards the data that gets inserted into multiple CFs and Drops the CFs as a whole when ttl has expired. Note that, when the Db is opened second time with a different ttl, the expiry time of the data is calculated based on the new ones.
Alternatively, you can create your own custom compaction filter to remove the old data in compaction time. This actually saves the pain of issuing a delete and can be done as a background job.
krithikaGopalakrishnan
- 1,217
- 10
- 28