45

I'm using redis lists and pushing to new items to a list. The problem is I really only need the most recent 10 items in a list.

I'm using lpush to add items to a list and lrange to get the most recent 10.

Is there anyway to drop items after a certain number? I'll end up with lists that may have 1,000's of items and can cause performance issues with latency.

Thank you!

dzm
  • 22,844
  • 47
  • 146
  • 226

6 Answers6

58

After every lpush, call ltrim to trim the list to 10 elements

See http://redis.io/commands/ltrim

Sripathi Krishnan
  • 30,948
  • 4
  • 76
  • 83
37

You can use LTRIM intermittently after any LPUSH, no need to call LTRIM after every LPUSH as that would add to overall latency in your app ( though redis is really fast, but you can save lots of LPUSH operations )

Here is a pseudo code to achieve an LTRIM on approximately every 5th LPUSH:

LPUSH mylist 1
random_int = some random number between 1-5
if random_int == 1:  # trim my list with 1/5 chance
   LTRIM mylist 0 10

Though your list may grow to be a few elements more than 10 elements at times, but it will surely get truncated at regular intervals. This approach is good for most practical purposes and saves a lot of LTRIM operations, keeping your pushes fast.

DhruvPathak
  • 42,059
  • 16
  • 116
  • 175
  • 15
    You can almost smell when a piece of advice comes straight from someone's experiences in production ;) – FeignMan Jan 24 '18 at 16:31
  • How can you be sure that getting a random number and checking if it is something isn't more expensive than an `LTRIM` call? – fiatjaf Apr 26 '18 at 17:30
  • 1
    @fiatjaf Mostly because of LTRIM being an over the network operation, however if both operations are part of a single Lua script, we might have to benchmark. However, I do strongly feel, that the random number operation is going to be simpler than LTRIM in both cases. – DhruvPathak Apr 28 '18 at 19:03
  • 2
    @DhruvPathak If one is using a language/library that supports pipelining, then it is not necessary to write a script to save the round trip time. – George Leung Nov 30 '18 at 07:51
12

The following code,

  • pushes the item to the list,
  • keep the size fixed to 10,
  • and returns the most recent 10 elements

in a transaction.

MULTI
LPUSH list "item1"
LTRIM list 0 9
LRANGE list 0 9
EXEC
ovunccetin
  • 8,443
  • 5
  • 42
  • 53
  • 3
    @PaulT.Rawkeen why? it can be achieved with Reids builtin commands (wrapped in transaction), I don't see a reason to make it a lua script. – Roy Miloh Apr 15 '18 at 19:07
11

No one has ever mentioned the real solution about storing only most 10 recent items.

Let's create a sample list with 15 items (here just numbers):

RPUSH list 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Now indicate offset from the end of the list:

LTRIM list -10 -1

Show list

LRANGE list 0 -1

 1) "6"
 2) "7"
 3) "8"
 4) "9"
 5) "10"
 6) "11"
 7) "12"
 8) "13"
 9) "14"
10) "15"

Now you can add new items and run trim:

RPUSH list 16
LTRIM list -10 -1

 1) "7"
 2) "8"
 3) "9"
 4) "10"
 5) "11"
 6) "12"
 7) "13"
 8) "14"
 9) "15"
10) "16"
kuzey beytar
  • 3,076
  • 6
  • 37
  • 46
7

Just an alternative. According to official doc of LPUSH, it returns the length of the list after the push operations. You can set a threshold length like k (in your case k > 10) and call LTRIM when returned length is bigger than k. Sample pseudo code as follows:

len = LPUSH mylist xxx 
if len > k:   
  LTRIM mylist 0 9
LRANGE mylist 0 9

It's more controllable than random method. Greater k triggers less LTRIM but with more memory cost. You can adjust k according to how often you want to call LTRIM since calling extra command is more expensive.

wei tu
  • 89
  • 1
  • 4
0

Calling LTRIM <list-name> -1 -10 after LPUSH <list-name> <item> is the simplest answer. Many had covered it.

You must do this two operations in a transaction or must use Lua script to ensure the operation is atomic.

giridhar
  • 141
  • 1
  • 7