Redis lua when to really use it?

Question

I've started to research and play a bit with lua and have found it to be great when wanting to grab ranges of keys. Ex:

business:5:visits:2013-11-12
business:5:visits:2013-11-13
etc

With lua I only have to send one command to redis instead of the complete range of dates.

Now I'm thinking about converting more of our logic and move it onto Redis.

Take our message storing process which currently looks like this:

// create a new unique id
redisClient.incr(Config.messageId, function(err, reply) {
    var messageId = reply.toString();
    var timestmp = Date.now();

    // push message
    redisClient.zadd(Config.history + ':' + obj.uid + ':' + obj.channel.replace(/\s+/g, ''), timestmp, messageId);

    // store the message data by messageId
    redisClient.hmset(Config.messageHash + ':' + messageId, {
        'user_id': obj.uid,
        'text_body': "some text",
        'text_date': timestmp,
    });


    // set expires
    redisClient.expire(Config.history + ':' + obj.uid + ':' + obj.channel.replace(/\s+/g, ''), Config.messageExpire);
    redisClient.expire(Config.messageHash + ':' + messageId, Config.messageExpire);


    // add to mysql-sync queue
    redisClient.RPUSH(Config.messageMySQLSyncQueue, Config.messageHash + ':' + messageId);

});

The above could easily be converted into lua, but is it worth it for performance?

Would it be faster to write this in Lua instead and only have to issue 1 command to Redis? Would it cause problems with blocking other commands?

Answer 1

Lua scripts are meant to work like MULTI commands. Actually most commands that you would develop using MULTI commands from a Redis client can be implemented in Lua. That is, you can encapsulate some complex operations in a script and your data layer will perform the atomic write operation without worrying about your data modeling strategy on Redis.

Also, I find them useful when you want to perform quick but complex read operations. For example, you might want to get objects in order. Objects are stored in a hash key while the order is defined by a sorted set key. You get a range of the so-called sorted set and you get objects in hash using hmget.

The most important point is Lua scripts should implement things that can execute as fast as possible, since Redis will block other operations while a Lua script is running. That is, you need to perform quick interruptions or your overall Redis performance will decrease a lot.

Arguments to not use Lua

I would argue that you should use them when you really need them. Usually, clients are developed using high-level programming languages like C#, Java, JavaScript, Ruby... and they provide better development experience: good debuggers, IDE, code-completion...

Summary: you should use them if you can prove that there's an actual benefit (in performance) if you turn some part of your domain logic into Redis Lua scripts.

Answer 2

There is one more drawback of Redis's Lua scripts: you can't issue operations to another instances of Redis even though they're on the same physical server. It's a common practice to run multiple instances of Redis on one server to utilize all the CPU cores. So you have to control many instances of Redis from a client.

To summarize everything that was said above:

1. Lua scripts are pure functions. You can't do outside requests like http requests or even requests to other instances of Redis. You can't make any side effects.
2. Lua scripts block everything. You can't run two Lua scripts in parallel. As a consequence you can't do any big job in Lua e.g. you can't run an infinite loop in Lua to do something in background. It's like to format a floppy disk on Windows 95 if you know what I mean :-)

In short, Redis is not an application server. So you cannot easily write any logic you want on Lua and be sure that everything is alright.

If you need a real application server inside a NoSQL database then try Tarantool for example. It also has Lua scripts, but the difference is that they do not block each other, they're still executed as a one ACID transaction (if you don't do external requests) and they may do any side effects you want, even issue requests to external databases like Tarantool or Redis or do any http requests. Which is achieved by executing each Lua script in a separate fiber and by row-based replication of all the changes that are done by Lua scripts.

Answer 3

Lua is nice and we used it in some situations - especially when we wanted some atomic operations to happen, but in your case you will have problems converting your above code to Lua script and run to Redis. And that's because of this line:

var timestmp = Date.now();

After a line like this you can't perform SET operations anymore, basically because how master-slave replication is handled in Redis. Take a look here: http://redis.io/commands/eval#scripts-as-pure-functions
This is like one more argument against using Lua scripts.

Answer 4

TL;DR: do not use a Lua script (for this)

Slightly longer: Redis' Lua scripting semantics call against generating key names via code and state that any keys used by a script should be provided as arguments (using the KEYS array).

Longer still: see quote from http://redis.io/commands/eval

All Redis commands must be analyzed before execution to determine which keys the command will operate on. In order for this to be true for EVAL, keys must be passed explicitly. This is useful in many ways, but especially to make sure Redis Cluster can forward your request to the appropriate cluster node

Answer 5

Lua scripts are really powerful. As you described it correctly, it allows to limit network roundtrips between the redis server and the client. Also, you don't send the script as String all the time, only the SHA1 should be sent after the first call, which is quite small.

There are best practices with Lua: if you will shard data or use replication, be sure to separate lua scripts that are "read-only" from those that actually write in the cluster (they must be executed on master).

Compute before your lua call, all the keys that you need in redis, also you don't access to time related vars natively in lua (embedded in redis), it means that time-related values have to be computed outside lua. Generally it is better to do most of the job outside lua, using it only to batch operations against redis and limit network activity.

Finally, be really careful with lua timeouts (you can overwrite it in redis). As a lua execution is blocking in a redis instance, it is a Stop-The-World operation so it must not be too long to execute (design your algorithms this way "Divide and Conquer") or everything will get slowed down.

Also threshold issues can appear: consider cleaning redis keys in a lua script. If it takes too long (too many keys / operations to process) it will fail due to lua timeout. Then, your data grows in redis due to activity. The next time you try to clean with lua, it would have to process even more keys in order to clean redis, as a result it has less chances to succeed! You could get an out-of-memory because of a lua timeout...