Laravel tagging overhead leaving behind significantly large reference sets using redis

Question

I am using Laravel 9 with the Redis cache driver. However, I have an issue where the internal standard_ref and forever_ref map that Laravel uses to manage tagged cache exceed more than 10MB.

This map consists of numerous keys, 95% of which have already expired/decayed and no longer exist; this map seems to grow in size and has a TTL of -1 (never expire).

Other than "not using tags", has anyone else encountered and overcome this? I found this in the slow log of Redis Enterprise, which led me to realize this is happening:

I checked the key/s via SCAN and can confirm it's a massive set of cache misses. It seems highly inefficient and expensive to constantly transmit 10MB back and forth to find one key within the map.

score 0 · Answer 1 · answered Jan 14 '23 at 06:29

This quickly and efficiently removes expired keys from the SET data-type that laravel uses to manage tagged cache.

use Illuminate\Support\Facades\Cache;

function flushExpiredKeysFromSet(string $referenceKey) : void
{
    /** @var \Illuminate\Cache\RedisStore $store */
    $store = Cache::store()->getStore();
    
    $lua = <<<LUA
    local keys = redis.call('SMEMBERS', '%s')
    local expired = {}
    for i, key in ipairs(keys) do
    local ttl = redis.call('ttl', key)
    if ttl == -2 or ttl == -1 then
        table.insert(expired, key)
    end
    end
    if #expired > 0 then
    redis.call('SREM', '%s', unpack(expired))
    end
    LUA;
    
    $store->connection()->eval(sprintf($lua, $key, $key), 1);
}

To show the calls that this LUA script generates, from the sample above:

10:32:19.392 [0 lua] "SMEMBERS" "63c0176959499233797039:standard_ref{0}"
10:32:19.392 [0 lua] "ttl" "i-dont-expire-for-an-hour"
10:32:19.392 [0 lua] "ttl" "aa9465100adaf4d7d0a1d12c8e4a5b255364442d:i-have-expired{1}"
10:32:19.392 [0 lua] "SREM" "63c0176959499233797039:standard_ref{0}" "aa9465100adaf4d7d0a1d12c8e4a5b255364442d:i-have-expired{1}"

Using a custom cache driver that wraps the RedisTaggedCache class; when cache is added to a tag, I dispatch a job using the above PHP script only once within that period by utilizing a 24-hour cache lock.

Here is how I obtain the reference key that is later passed into the cleanup script.

public function dispatchTidyEvent(mixed $ttl)
{
    $referenceKeyType = $ttl === null ? self::REFERENCE_KEY_FOREVER : self::REFERENCE_KEY_STANDARD;

    $lock = Cache::lock('tidy:'.$referenceKeyType, 60 * 60 * 24);

    // if we were able to get a lock, then dispatch the event
    if ($lock->get()) {
        foreach (explode('|', $this->tags->getNamespace()) as $segment) {
            dispatch(new \App\Events\CacheTidyEvent($this->referenceKey($segment, $referenceKeyType)));
        }
    }

    // otherwise, we'll just let the lock live out its life to prevent repeating this numerous times per day
    return true;
}

Remembering that a "cache lock" is simply just a SET/GET and Laravel is responsible for many of those already on every request to manage it's tags, adding a lock to achieve this "once per day" concept only adds negligible overhead.

Laravel tagging overhead leaving behind significantly large reference sets using redis

1 Answers1