3

I'm writing some logic for redis inside lua and almost each of my scripts have something common, it would be really handy to move out this to the shared function but

  1. redis can't use lua's require statement
  2. officially you can't call other redis function(see: https://stackoverflow.com/a/22599862/1812225)

For example I have this snippet literally everywhere

local prefix = "/" .. type
if typeId then
    prefix = prefix .. "(" .. typeId .. ")"
end

I'm thinking about some post-processing before feeding scripts to redis but this seems like an over-kill...

What is the best practice to solve/reduce this problem?

Updated:

local registryKey = "/counters/set-" .. type
local updatedKey = "/counters/updated/set-" .. type
if typeId then
    redis.call("SAdd", updatedKey, name .. ":" .. typeId)
    redis.call("SAdd", registryKey, name .. ":" .. typeId)
else
    redis.call("SAdd", updatedKey, name)
    redis.call("SAdd", registryKey, name)
end

is another code sample and it can't be trivially moved to client-side as it invokes redis commands, and works as a part of transaction

Thanks!

Community
  • 1
  • 1
let4be
  • 1,048
  • 11
  • 30
  • 1
    Assume for a moment that it is possible to have this common snippet "shared" - would you really want to use it? Put differently, here's a mundane task that needs to be done a lot of times... Pushing it into a Lua script means you'll be using resources from your Redis database to perform something that could be done elsewhere. I'm not saying it is wrong, but as a rule of thumb I try to keep as this type of logic outside the database whenever possible. – Itamar Haber Aug 05 '15 at 13:56
  • Regarding this one snippet you are probably right @itamar-haber. But I have another case when I need to do thing with conditional logic which triggers redis commands over and over again... It can't be easily moved to client side – let4be Aug 05 '15 at 14:24
  • On the other hand I probably should just create another script with repetative steps and use them both inside a `MULTI`, i.e. `MULTI EVALSHA EVALSHA ...` This just seems non ideal from the point of bandwidth but shouldn't be a big issue – let4be Aug 05 '15 at 14:28
  • 1
    Each case is different :) basically the options are a) copy-paste b) use the undocumented f_ approach or c) "hack" around Redis' protection of global. I feel that the safest is (a) – Itamar Haber Aug 05 '15 at 14:29
  • @ItamarHaber two things. a) what is the undocumented f_ approach? and b) Why the trick used by redis-lua-debugger be used here? (https://github.com/RedisLabs/redis-lua-debugger) (of course, taking into account the limitation regarding slaves not running the script by themselves). – Ignacio Aug 06 '15 at 16:05
  • Ok, added a real answer :) – Itamar Haber Aug 06 '15 at 18:06

1 Answers1

4

"Hack" #1

After you SCRIPT LOAD something, you get back a sha1 hash that you can use with EVALSHA. The same sha1 value can be used to call that script from inside another script - simply call the function f_<sha1>. That said, there are some differences in how you pass the KEYS/ARGV structures when used that way.

Note that this is undocumented behavior, which means the behavior could change in a future version of Redis.

Credit for teaching me this goes to Dr. Josiah Carlson who, in turn, credits someone else (IIRC Fritzy). For more information check out his lua-call Python wrapper: https://github.com/josiahcarlson/lua-call

"Hack" #2

Redis sandboxes Lua and puts several restrictions on it in order to maintain sanity. You could go around some of these, e.g. access _G and define your utility function there so it will be available to all scripts (like I did with https://github.com/redislabs/redis-lua-debugger).

However, this is also pretty risky - besides potential replication issues, this usage is untested and could therefore lead to undefined behavior (I managed to crash quite a few instances with my little script ;)).

P.S.

Both hacks require additional administrative work to ensure that these "global" scripts are actually loaded before any other script calls them.

Itamar Haber
  • 47,336
  • 7
  • 91
  • 117
  • Thanks for the detailed response. I wonder if eventually a standardized way to have utility code available on Redis instances will be added. I foresee myself using some code over and over in many scripts (redis.log wrappers, libraries like underscore.lua and the like, etc). – Ignacio Aug 07 '15 at 18:02
  • Suggestions/ideas/PR contribs are always welcome in the Redis community - why not present your case and start the discussion? – Itamar Haber Aug 07 '15 at 19:07
  • Is Hack #1 actually using undocumented behaviour? The sha1 is *the* way a client would get a handle on a loaded script. (Is it calling f_ that's the problem?) – Tim Barrass Oct 15 '15 at 09:21
  • 1
    @tim exactly, the f_sha1 is an internal implementation and you can't count on it to remain unchanged in future versions. – Itamar Haber Oct 15 '15 at 10:54