MIPS, ELF and partial linking

Question

I have a big software project with a complicated build process, which works like this:

1. Compile individual source files.
2. Partially link object files for each module together into another .o using ld -r.
3. Hide private symbols in each module using objcopy -G.
4. Partially link module objects together, again using ld -r.
5. Link modules together into a shared object.

Step 3 is required to allow module-private global variables that aren't exported to the rest of the project.

This all works fine with ARM and IA32. Unfortunately, now I have to make things work on mips (specifically, mipsel-linux-gnu for Android). And the MIPS shared object ABI is significantly more complex than on the other platforms and it's not working.

What's happening is that step 5 is failing with this error:

CALL16 reloc at 0x1234 not against global symbol

This seems to be because the compiler generates CALL16 relocations to call functions in another compilation unit, but CALL16 only allows you to call global symbols --- and because of step 3, some of the symbols that we're trying to call aren't global any more.

At this point I can see several possible options:

• persuade the linker to resolve the CALL16 relocations to normal intra-compilation-unit PC-relative calls at step 2.
• ditto, but at step 4 or 5.
• tell the compiler not to generate CALL16 relocations for inter-compilation-unit function calls.
• other.

Disabling step 3 is, I'm afraid, not an option due to external requirements.

What I'd really, really like to do is to generate absolute code which gets patched at load time to the right addresses; it's smaller, much faster, and vastly simpler, and we don't need to share the library between processes. Unfortunately it appears that Android's dlopen() doesn't seem to support this.

Currently I'm out of my depth. Anyone have any suggestions?

This is gcc 4.4.5 (from Emdebian), binutils 2.20.1. Target BFD is elf32-tradlittlemips. Host OS is Linux, and I'm cross-compiling for Android.

Addendum

I am also getting warnings like this from step 4.

$MODULE.o: Can't find matching LO16 reloc against `$SYMBOLNAME' for R_MIPS_GOT16 at 0x18 in section `.text.$SYMBOLNAME'

Looking at the disassembly of the input to step 4, I can see that the compiler's generated code like this:

50:   8f9e0000        lw      s8,0(gp)
                      50: R_MIPS_GOT16        $SYMBOLNAME
54:   8fd9001c        lw      t9,28(s8)
58:   0320f809        jalr    t9
5c:   00a02021        move    a0,a1

Doesn't GOT16 fix up to the high half of an address, and should be followed with a LO16 for the low half? But the code looks like it's trying to do a GOT indirection. This puzzles me. I've no idea if this is related to my earlier problem, or is a different problem, or is not a problem at all...

Update

Apparently MIPS simply does not support hidden global symbols!

We've gotten around it by mangling the names of the symbols that are supposed to be hidden so that nobody can tell what they are. This is pushing the external requirements quite a lot, but I sold management on it by pointing out that it was the only way to get a shippable product.

That's totally gruesome (and involves some deeply disgusting makefile work to do), so I'd rather like a better solution, if anyone has one...

Answer 1

I'm not sure about about the specific GOT issues you are having. There are a lot of bugs and issues with GOT, LO16/HI16 stuff in binutils. I think most have been fixed in the version your using, unless you are targeting MIPS16 (which you don't seem to be doing). LO16 is really only necessary there, beyond MIPS16 you're pulling the full 26-bit offset out of the GOT since you have 32-bit registers. LO16 isn't needed, but is still formally required by some ABI/APIs but it was fudged to be at most an warning (you may try removing a -Werror at that phase if you are using it). I only understand the very basics of that part honestly, the rest of your situation I had some recommendations on though, if not an answer (hard to be sure given the complexity of your setup).

In MIPS (and most assemblies I'm familiar with) you have your basic three levels of visibility: local, global, and weak. In addition you have comm for shared objects. GNU, of course, likes to have things more complicated and adds more. gas provides protected, hidden, and internal (minimally, it is hard to keep up with all the extensions). With all of this the steps your setting in manually fiddling around with visibility seem unnecessary.

If you can remove the intermediate globalness of the variables, it should remove you need to make them unglobal, which can only serve to simplify any GOT issues you run into later.

The overall problems is a bit confusing. I'm not sure what you mean by hidden global symbols, it's a bit a contradiction (of course portability and specific projects give crazy problems and restrictions). You seem to want cross assembly unit symbols at one stage, but not a later stage. Without using GNU extensions (something best avoided in my book), you may want to replace the globals in steps 1-2 with comm and/or weakglobals. You could always use use preprocessor trickery to avoid having multiple sub-units at the stage even (ugly, but that's portable code at this level).

You really have a setup of 1) sub-modules 2) sub-modules -> modules 3-5) modules -> shared library. Simplifying that can't hurt. You can always interpose at 2) or 3-5) a C-level interface just to find what assembly GCC will product for you architectures and use that as a basis for breaking visibility up into clean interfaces.

Wish I could give you a tailor made solution, but that's pretty impossible without your full project to work from. I can reassure that while MIPS location (especially the toolchains) have issues, the visibility options (especially if you are using gas, libbfd, and gcc) are the same.

Answer 2

your binutils is too old. some changesets in 2.23 may resolve your problem, like "hide symbols without PLT nor GOT references".