1

UPDATE I split the file into multiple files each roughly with 1.5 million lines and no issues.

Attempting to pipe into Redis 6.0.6 roughly 15 million lines of SADD and HSET commands properly formatted to Redis Mass Insertion but it fails with the following message:

ERR Protocol error: too big mbulk count string

I use the following command:

echo -e "$(cat load.txt)" | redis-cli --pipe

I run dbsize command in redis-cli and it shows no increase during the entire time.

I can use the formatting app I wrote (a c++ app with client library redis-plus-plus), which correctly formats the lines, write to std::cout then using the following command as well:

./app | redis-cli --pipe

but it exits right away and only sometimes produces the error message.

If I take roughly 400,000 lines from the load.txt file and load it in a smaller file then use echo -e etc.... it loads fine. The problem seems to be the large number of lines.

Any suggestions? It's not a formatting issue afaik. I can code my app to write all the commands into Redis but mass insertion should be faster and I'd prefer that route.

Shaun Ingoldsby
  • 11
  • 2
  • Why are you using `echo` instead of just `cat`, as in the documentation? My guess is that instead of streaming the output, this way forces the shell to read the entirety of the file and then pass the whole thing to `redis-cli`, which then chokes on large sizes. – Kevin Christopher Henry Aug 06 '20 at 09:33
  • If I use cat to send the lines of text to redis then the carriage return and new line \r\n are not formatted properly. Echo allows them to be read correctly for some reason. I read that bash has an issue with this and this was the solution. – Shaun Ingoldsby Aug 06 '20 at 15:21
  • OK, but as I said I think that is your problem, so I suggest you figure out why using `cat` is not working and find a different solution. Perhaps your `cat` is aliased to something else, like `cat -v` (use `type cat` to find out)? And there are other ways to send a file to stdout. – Kevin Christopher Henry Aug 07 '20 at 01:33
  • I am also having same issue with cat \n\r converted to new line – Kishor Patidar Jun 23 '23 at 07:54

0 Answers0