Keeping everything in a single lua bytecode chunk?

Question

I've embedded lua together with a bytecode chunk into a project written in C. Now when I extend my lua code base by adding .lua files, is there a way to keep this code in a single bytecode chunk?

(I know how to load multiple bytecode chunks. But making it load a single chunk and then forgetting about the glue code would just seem comfortable.)

I tried to use textual inclusion, but it seems there's no keyword for this in Lua. "Require" and "dofile" look at the files at run time, so the resulting bytecode after running "lua -b ..." won't include the code of those files.

And there's no way for combining bytecode files either, is there? I mean so that, when creating a bytecode file, the "require" command would add the code of all those files into one bytecode file.

PS: Michal Kottman's answer works for Lua, which is what I asked for. I thought Lua and LuaJIT would work the same way. They don't. To combine multiple .lua files to one LuaJIT bytecode file, should one

• use "LuaJIT -b" (seems not to work)
• compile Lua's luac.c with LuaJIT sources
• emulate luac.c with lua commands (without C API) ?

Answer 1

You can combine multiple files into a single file using luac. When run, all the chunks from the source files are executed in the order they were added to the compiled file:

$ echo "x=1"         > l1.lua
$ echo "y=2"         > l2.lua
$ echo "print(x, y)" > l3.lua
$ luac -o run.luac l1.lua l2.lua l3.lua
$ lua run.luac
1   2

You can load this file into Lua from C using luaL_loadfile, which places a function on the top of the stack if it loaded succesfully. Then you can just run this function using lua_call to run all the combined compiled files.

Note that you can embed the contents of the compiled file as a string into your project, no need to keep it in external file.

Update for LuaJIT 2

As you have found, you can use the Lua Compiler in Lua to get a combined file which can be loaded as previously noted. This is a simplified version, which outputs to stdout:

-- http://lua-users.org/wiki/LuaCompilerInLua
-- compile the input file(s) passed as arguments and output them combined to stdout
local chunk = {}
for _, file in ipairs(arg) do
  chunk[#chunk + 1] = assert(loadfile(file))
end
if #chunk == 1 then
  chunk = chunk[1]
else
  -- combine multiple input files into a single chunk
  for i, func in ipairs(chunk) do
    chunk[i] = ("loadstring%q(...);"):format(string.dump(func))
  end
  chunk = assert(loadstring(table.concat(chunk)))
end
io.write(string.dump(chunk))

For the previous sample, you can use it as follows:

$ luajit combine.lua l1.lua l2.lua l3.lua > out.ljc
$ luajit out.ljc
1   2

Answer 2

Another alternative is to use a tool like Mathew Wild's squish to collect all your Lua sources into a single .lua file. One nice feature of squish is that it supports a variety of filters to help make the squished package smaller than the total of the source files.

After applying squish, you could run the result through luac to get bytecode. However, bytecode is often larger than the source code, and almost certainly larger if a more aggressive filter (such as gzip) is used in squish.

If your bytecode file were stored separately from the executable (or worse, transmitted over the network), I'd also bring up security concerns related to bytecode that don't apply to Lua source code. However, bytecode linked in to an application is much harder to subvert since harmful bytecode will never be the result of running luac.

Answer 3

After tried luac, luajit and squish, I found all of them requires you to maintain a file list to combine. That is tiresome when working on a project contain many lua files.

So I wrote a little tool to merge lua files by analyzing requires

Here it is: https://github.com/yi/node-lua-distiller

Hope it will helpful.

Answer 4

loadstring is what your looking for. It takes a string and loads if (after compiling it to byte code if necessary). It results in a function that can be executed to run the contained code afterwards.

If you still need something more general, take a look at load which lets you specify a function to feed the chunks in.

Both functions can process both Lua source code and compiled byte code.