For general comparison of bytecode interpreters and abstract syntax tree interpreters, see Wikipedia:Interpreter.
Lua has been using a virtual machine since its birth, and introduced a register-based virtual machine in Lua 5.0 (which is the first register-based virtual machine to have a wide use.) The Evolution of Lua, written by the authors of Lua, provides lots of resources on that topic.
Another paper written by the authors of Lua, The Implementation of Lua 5.0 provides lots of details on the implementation, Chapter 7 The Virtual Machine would be especially helpful to you, this paper is also a great reference for reading Lua's source code.