0

Background of problem

Hello all , i have made a project in golang gin , and I have integrated the redis Clusterclient in it using "github.com/go-redis/redis/v7"

P.S. Redis that I am using is a redis cluster hosted on AWS

the redis commands that I am using are simply redis.Get and redis.Set only.

Now I have made one API and used caching in it, and when I run it locally, response times are around 200 to 300ms, which is awesome (thanks to Redis)

Main Problem

now when I start doing the load testing on the same API with around 100 concurrent users , response time gets significantly increased ( around 4 seconds). I used spans to monitor the time taken by a different part of the code, and I got this

enter image description here

Getting from primary, getting from secondary are for the redis.Get command

Setting the primary , setting the secondary are for redis.Set

both commands are taking around 1 sec to execute, which is unacceptable,

can anyone please tell me some way, so that I can tackle this problem and reduce the time for the redis commands to execute

kostix
  • 51,517
  • 14
  • 93
  • 176
iron_man83
  • 67
  • 2
  • 9
  • 2
    You need to measure CPU, RAM, disk and network utilization to figure out which one is the bottleneck. Also 200 to 300 ms latency is actually already quite high unless there is a large physical distance between the database and your client. – Zyl Jan 26 '22 at 09:44
  • Please note that it's not a problem about programming in Go, and hence must not be tagged `go`. If you were to trace this perf problem to a particular place (or places) in the indicated Redis client library's code, then feel free to create a question tagged `go` asking how to improve those places. – kostix Jan 26 '22 at 10:04

1 Answers1

0

Ok so I have solved this somehow. Firstly I have updated my golang redis client library from go-redis/v7 to go-redis/v8 . And it made a significant improvement. I will advise everyone to do the same.

But still I am suffering from high response time , so the next step for me wa sto change the redis infra. Earlier I was using a redis cluster which only had 1 shard, but now I have moved to another redis having 4 shards..

And it made a huge difference , my response goes from 1200ms to 180ms. Kindly note that these response time are coming when I am doing a load testing with 100 concurrent users with an average of about 130rps

So in short upgrade your redis client , upgrade your redis infra

iron_man83
  • 67
  • 2
  • 9