2

I have used the command line below to backup a key into a file . The file size is 1.5GB . It is a hash key with many many fields .

redis-cli --raw dump mykey | head -c-1 > myfile

When I used the command line below to restore the key , I always got " Connection reset by peer ", I changed to use redis python client , got the same error as "Error 104 while writing to socket. Connection reset by peer." When I changed to a 500MB hash file with one key for same operation , it got no problem and successfully restored . I have set max memory for redis is 9GB and RAM is 14GB . Redis database size is 1.5GB .

cat myfile | redis-cli -x restore mynewkey 0

I guess it might be redis limitation for 1GB input buffer problem .

Is there any solution I can either reduce the dumped file size when redis dump or increase the input buffer to allow file more than 1GB .

Richard Mao
  • 202
  • 3
  • 12
  • The [manual on data types](https://redis.io/topics/data-types) says that a string can be 512 MB – apokryfos Sep 09 '17 at 06:49
  • I edited the question , it is a hash key with many fields , one is 1.5GB and the other is 500MB . 1.5GB is not working while 500MB is okay . – Richard Mao Sep 11 '17 at 06:15
  • That kind of reinforces the statement that "strings can only be 512MB" – apokryfos Sep 11 '17 at 08:42
  • in the manual on data types , it says "Every hash can store up to 232 - 1 field-value pairs (more than 4 billion)" , not ever mentioned the length limitation . – Richard Mao Sep 11 '17 at 08:46
  • 1
    The limitation is more fundamental than that. You can't have any string longer than 512 MB no matter where it is. Just make sure you don't have a string that goes above that limit as either a key or a hash value – apokryfos Sep 11 '17 at 08:47
  • i did not have problem while using 1.5GB hash key in redis instance(i guess it should be even bigger size since 1.5GB is dumped file size ) , and i did not have problem when dumped the 1.5GB hash key to file . The only problem I had is restore 1.5GB hash key dumped file into redis instance again . So , i guess it is input file size limitation . – Richard Mao Sep 11 '17 at 08:52
  • Sounds like a timeout issue, redis-cli has a hard coded TO, I do not remember the value but you can find it in the sources easily. Python redis-py has is configurable, try raising it. FYI, in hash is constructed from strings, the 512MB limit is per string in the hash. – Ofir Luzon Sep 11 '17 at 16:14

0 Answers0