A problem occurred when using cluster on redis version 3.0.7,
In our cluster, there exist many huge keys, entry numbers of some keys, whose size is large to 500M bytes, have reached millions.
Now we want to scale out the cluster using command migrate, but this command is synchronous, which may lead to source and destination node block.
Is there any elegant way to migrate these keys?
According to my experiments, the response time (RT) grew rapidly, with an increased length of the key.
In this experiment, my aim is to observe how response time (RT) changed according to the key length. I created a JavaScript script that reads all keys and calculates the RT. For the purposes of my experiment, I loaded 1000, 5000 and 10000 keys in node. The results are represented in below Figure.
I observed that the RT grew rapidly, with an increased length of the key. In detail, for a node with 10000 keys and size of key 10, the RT was 91,7 ms and for the same node and key length 24000, the RT was 1014,3 ms. Furthermore, to retrieve one single key with size 10 the RT was 0.0127 and to retrieve one single key with size 24000 the RT was 0,102.
So, when bulk loading and replication aren’t available you can try to migrate one key at a time with MIGRATE. The MIGRATE command atomically transfers a key from a source database to a destination database. But this is a destructive action unless the COPY option is passed (COPY -- Do not remove the key from the local instance).
Starting with Redis 3.0.6 MIGRATE supports a new bulk-migration mode that uses pipelining in order to migrate multiple keys between instances without incurring in the round trip time latency and other overheads that there are when moving each key with a single MIGRATE call.
In order to enable this form, the KEYS option is used, and the normal key argument is set to an empty string. The actual key names will be provided after the KEYS argument itself, like in the following example:
MIGRATE 192.168.1.34 6379 "" 0 5000 KEYS key1 key2 key3
So, to conclude, there is no any elegant way to migrate keys from source to destination, apart from the afforementioned.
One possible solution is to isolate the "large" keys in one or more nodes and set up a threshold concerning the key length. If a key exceed this threshold then keep the key in the current node. If not, you can migrate in different node or DB. But you must conduct some experiments in order to set up this threshold and observe what means "large", according to response time and possible SLAs.
You can check also this: https://fnordig.de/2014/03/11/redis-cluster-with-pre-existing-data/
