0

I am running a Redis instance with maxmemory and usage of around 25GB. It is being run as Statefulset in Kubernetes. As the redis-pod can get scheduled to any of the boxes, and can get restarted any time I AOF backup over RDB.

But, yesterday the redis-pod got restarted and it took around 5 min to load the data, which made me think if the RDB backup is better suited if the data is large?

  • I know, that the AOF file size can outgrow, and is automatically rewritten to optimize.
  • But even in a 100% optimized state, if the DB has to restart, IMO it will take the whole time to call all the commands which will fill up the data back to 25GB, and which would be time-consuming. (around 5 mins, which I monitored)
  • I want to reduce this time and thinking of going back to RDB snapshots, though the data will be lost, that's a tradeoff for fast startup time. Will schedule the BGSAVE for a shorter time, to avoid or reduce the data loss.

I wanted to know, if it is preferable to use RDB backup if the data size is large and you want to avoid the slower startup time? And if running BGSAVE every 5 or 10 mins, if the data size is large (above 15GB) is good practice or not?

(also the large data size may not be large enough for others, but this it's about the startup time, I am worried about, and the downtime it brings with that)

kadamb
  • 1,532
  • 3
  • 29
  • 55
  • 1
    A BGSAVE every 5 minutes is nothing out of this world, i think you can aim a lot lower since you also configure a threshold for changed keys. All of this greatly depends on your specifics ofc. Another thing you can give a try is a combination of AOF and RDB, specifically `AOF with RDB-preamble`. There's a clear [lack of documentation](https://github.com/redis/redis-doc/issues/908) but you can find some resources [here](https://stackoverflow.com/questions/63436459/redis-load-from-rdb-and-keep-writing-the-aof) and [there](https://www.memurai.com/blog/redis-persistence-deep-dive). – nitrin0 Sep 30 '21 at 10:30

1 Answers1

1

I am the OP.

Not an answer, but wanted to share some data I collected after some experiments

  • I populated the Redis DB with 16.5GB of data, using the following command DEBUG POPULATE 165000000 PHPREDIS_SESSION

  • Then I rewrite/optimise the AOF file with the help of - BGREWRITEAOF. It took 209 seconds for the whole operation:
    aof_last_rewrite_time_sec:209

  • Just after the rewrite, I restarted the Redis instance, and it took 3minutes to restore the whole data to Redis memory of 16.5Gb

  • After this, I turned off the AOF and enabled the RDB backup, and called the BGSAVE to start backing up. It took 202 seconds for it to complete:
    rdb_last_bgsave_time_sec:202

  • After the RDB backup, I again restarted the Redis instance, and it again took around 3 minutes, more or less the same as AOF, for the whole data-set to restore in Redis memory.

Looks like the RDB and 100% optimized AOF file will take more or less the same time to restore the data.

It might take more time for an AOF, if it is not optimised or rewritten, as that many more commands will be run by Redis to restore data.

I also tested the startup time of a Redis instance with 25GB of data, and AOF enabled, it took 4m30s to restore the data.

Note:
The data presented above is taken with a single iteration only, and vanilla Redis without any tuning. The numbers might mismatch with some other situation or configuration. Just wanted to share my findings.

kadamb
  • 1,532
  • 3
  • 29
  • 55
  • Hey- thanks for sharing. So, considering that both methods incur about the same startup time then would you say then that AOF (optimized) is the way forward? – skupjoe Mar 12 '22 at 05:44
  • Also, did you experiment with `AOF with RDB-preamble`, as nitrin0 suggested? – skupjoe Mar 12 '22 at 05:45
  • Yes, we went with AOF-optimized, as we are running the redis as container on Kubernetes, it could get rescheduled anytime, we get more persistence with AOF. – kadamb Mar 16 '22 at 12:25
  • Nope, didn't try `AOF with RDB-preamble` – kadamb Mar 16 '22 at 12:25