Spring Data Redis bulk save using repository too slow, how to pipeline repository saves

Question

we are saving 250k Objects in Redis using Spring Data Redis Repositories (repository.save(List)). its too slow. its taking around 30 minutes.

i am using redisTemplate with pipeline to save 250k objects with same redis data structure as repository (like using hmset to store object, sadd to to store key. so that we can retrieve object using repository) save, managed to save 250k in 150 seconds.

how can we save objects using repository save with pipeline? so that i don't need to manually do stuff like hmset, sadd (this is the internally happens when repository saves data in redis)

Been having the same issue.. have you resolved this? Doing `.save()` in a pipeline? — Ken Flake, Mar 17 '20 at 03:30

score 1 · Answer 1 · answered Mar 17 '20 at 10:14

been having this same problem for days now, thought I just share what I did on my end to make ends meet.

So I gave up on using @Repository to save my DTOs on Redis via Redis Hash..

I was kind of forced to use commands like hSet inside the pipeline.

As of now you can't do a bulk save + pipeline using @Repository interfaces..

Also, Lettuce's logging every after Redis command also affects writing operations on dev environment.

Try switching off io.lettuce.core DEBUG logs and see if operations speed up.

I can share a code snippet on my situation if needed.

Spring Data Redis bulk save using repository too slow, how to pipeline repository saves

1 Answers1