Storing nested tables in nginx shared memory

Question

I am working with open-resty and lua to create a server for redirecting requests. Redirects are done based on some data from a lua datatree structure (nested tables)

I am looking for a way to populate this data once on startup and after that share the data between workers.

ngx.ctx can save arbitrary data but lasts only during request. Shared dict lasts till the end but can only save list of primitives.

I've read that it is possible to share data across lua modules. Because modules get instantiated only once at startup. The code is something like this

local _M = {}

local data = {
    dog = {"value1", "value4"},
    cat = {"value2", "value5"},
    pig = {"value3", "value6"}
}


function _M.get_age(name)
    return data[name]
end

return _M

and then in nginx.conf

location /lua {
    content_by_lua_block {
        local mydata = require "mydata"
        ngx.say(mydata.get_age("dog"))
    }
}

Is this third possibility thread safe? Is there something else which can achieve this?

There is not a lot of documentation on this, that is why posted it here. Any info would help, Thank you

piotrp · Accepted Answer · 2019-06-10T10:04:22.457

2

You can populate your data in init_by_lua, and access it later on. In your case initialization of mydata module can be achieved by:

init_by_lua_block {
     require "mydata"
}

init_by_lua runs once during nginx startup, and then the process it run in forks into workers, so each of them contain an independent copy of this data.

Workers are single-threaded, so you can safely access your data.

Now, if you want to modify your configuration at runtime, without reloading nginx, then it gets a bit more complicated. Each worker is independent, but we can use ngx.shared.DICT to propagate changes. Depending on your requirements there are two solutions you can use:

After each change put your configuration into shared dictionary. Create a timer that periodically reloads worker's configuration from this shared cache.
After each change put your configuration into shared dictionary, along with current timetamp or version number. On each request in a worker check whether this timestamp/version is never than the one cached locally - if it is then deserialize this configuration and cache it locally.

If you have an API that should be usable then you can use lua-resty-lock to create cross-worker critical sections that synchronize modification.

edited Jun 10 '19 at 10:04

answered Jun 08 '19 at 20:32

piotrp

3,755
1
24
26

Thank you for answering. It helps and I can use it. So, this would mean that if I want to update the date I would have to reload nxing. One more thing, I am interested in your opinion. Could you please take a look at this. Just the paragraph called "Module Instances" with code. (http://kiki.to/blog/2014/04/11/rule-4-make-stateless-modules/) Would this be a better solution? I would like to have an option of creating, updating , deleting values from the main data and for all workers to be able to access it. – djulb Jun 08 '19 at 21:41
1

I extended my answer to cover changing your data. Modules are good because they allow you to keep your code readable, so go ahead and use it. – piotrp Jun 10 '19 at 10:05
The data is stored in a nested table and shared.DICT doesn't accept complex data like nested tables. Shared dict can save map but only if keys and values are primitive. However you have a good point, i could have a serialized string, and a version in shared dict. A timer can trigger workers to deserialize data and use it. Thank you – djulb Jun 10 '19 at 10:36
1

[cjson](https://github.com/openresty/lua-cjson/) should be ok to serialize your data – piotrp Jun 10 '19 at 10:49

Storing nested tables in nginx shared memory

1 Answers1