Why does UPX break compiled SBCL apps?

Question

This is mostly a stupid question, since UPX (a tool that wrings extra bytes out of your executable files) saves a tiny amount of space over the built in compression in the buildapp tool.

A very small demo application creates a 42 megabyte file. Understandable, since the SBCL environment isn't tiny.

Passing the --compress-core option to buildapp shrinks that down to 9.2MB.

I thought I'd try throwing UPX at the resulting binary, and the savings only amounts to a few more bytes: 9994288 -> 9871360

However, the resulting file no longer runs anymore - it just jumps into the SBCL REPL (with no errors, it's as if I just ran sbcl by hand), and some poking around there reveals that the functions making up my test program no longer exist.

What did UPX do to the binary that resulted in this breakage?

Answer 1

This may not be the answer, but it may serve as a clue: I've found that if you add anything, even a single byte, to the end of an SBCL executable created with sb-ext:save-lisp-and-die, then all the definitions disappear, just as you described.

Perhaps SBCL creates executables by appending the core (which contains your definitions) to a copy of the SBCL ELF (or PE on Windows) binary, plus some metadata at the end so that SBCL can still find the beginning of the core even though it's appended to an executable.

If you hex-edit an executable created with save-lisp-and-die, you'll find that it ends with the string "LCBS" (SBCL backwards), which seems to support my theory. "LCBS" probably serves as a magic number, letting SBCL know that yes, this executable contains its own core.

UPX compresses executables, probably including that magic number at the end. When SBCL opens its UPX-compressed self on disk, it won't find "LCBS" at the end, so it assumes that there is no core appended to the executable.

I can't explain why the standard library seems to still be there if this is the case. It could be that SBCL loads /usr/lib/sbcl/sbcl.core (or its equivalent on Windows) in that case. This could be tested by moving the executable to a machine where SBCL is not installed and seeing if it works at all, and if so, whether you still have car, cdr, list, etc.