MIPS cross compilation error: illegal instruction

Question

I want to cross-compile C src for MIPS machine, but the compilation fails. Executing cross compiled binary on Target machine outputs "illegal instruction". Could you please let me know how to compile C src for MIPS machine?

Details are as follows:

Executing "readelf -h host.bin" (host.bin is binary program running on HOST machine) outputs:

ELF Header:
Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
Class:                             ELF32
Data:                              2's complement, little endian
Version:                           1 (current)
OS/ABI:                            UNIX - System V
ABI Version:                       0
Type:                              EXEC (Executable file)
Machine:                           Intel 80386
Version:                           0x1
Entry point address:               0x8048aac
Start of program headers:          52 (bytes into file)
Start of section headers:          8628 (bytes into file)
Flags:                             0x0
Size of this header:               52 (bytes)
Size of program headers:           32 (bytes)
Number of program headers:         9
Size of section headers:           40 (bytes)
Number of section headers:         30
Section header string table index: 27

Executing "readelf -h target.bin" (target.bin is binary program running on Target machine) outputs:

ELF Header:
Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
Class:                             ELF32
Data:                              2's complement, little endian
Version:                           1 (current)
OS/ABI:                            UNIX - System V
ABI Version:                       0
Type:                              EXEC (Executable file)
Machine:                           MIPS R3000
Version:                           0x1
Entry point address:               0x403570
Start of program headers:          52 (bytes into file)
Start of section headers:          114040 (bytes into file)
Flags:                             0x5, noreorder, cpic, mips1
Size of this header:               52 (bytes)
Size of program headers:           32 (bytes)
Number of program headers:         6
Size of section headers:           40 (bytes)
Number of section headers:         23
Section header string table index: 22

Executing "readelf -h cross_compiled.bin" outputs (cross_compiled.bin is binary that is cross compiled (by me) for Target machine):

ELF Header:
Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
Class:                             ELF32
Data:                              2's complement, little endian
Version:                           1 (current)
OS/ABI:                            UNIX - System V
ABI Version:                       0
Type:                              EXEC (Executable file)
Machine:                           MIPS R3000
Version:                           0x1
Entry point address:               0x400170
Start of program headers:          52 (bytes into file)
Start of section headers:          16444 (bytes into file)
Flags:                             0x1007, noreorder, pic, cpic, o32, mips1
Size of this header:               52 (bytes)
Size of program headers:           32 (bytes)
Number of program headers:         5
Size of section headers:           40 (bytes)
Number of section headers:         21
Section header string table index: 20

For the cross compilation, I have used buildroot which supports MIPS1 cross compilation.

I expected cross_compiled.bin is executed normally on Target machine. However, when I execute "cross_compiled.bin" on Target machine, it outputs "illegal instruction".

Could you please let me know how to solve the above problem? Since I am new to cross compilation, I cannot find what the exact reason of the problem is.

Any comments would be appreciated.

UPDATE:

On the target, executing "cat /proc/cpuinfo" outputs:

system type             : Broadcom BCM5354 chip rev 3
processor               : 0
cpu model               : BCM3302 V2.9
BogoMIPS                : 237.56
wait instruction        : no
microsecond timers      : yes
tlb_entries             : 32
extra interrupt vector  : no
hardware watchpoint     : no
VCED exceptions         : not available
VCEI exceptions         : not available
unaligned_instructions  : 1
dcache hits             : 0
dcache misses           : 0
icache hits             : 0
icache misses           : 0
instructions            : 0

On the target, executing "cat /proc/version" outputs:

Linux version 2.4.20 (ronger@ronger-linux.sh.deltagroup.com) (gcc version 3.2.3 with Broadcom modifications) #1 Tue May 22 17:40:19 EDT 2007

Command line I used for cross-compilation (cross_compiled.bin):

mipsel-linux-gcc -static -o cross_compiled.bin cross_compiled.c

cross_compiled.c:

#include <stdio.h>
int main()
{
  printf("Hello World\n");
  return 0;
}

Since my cross_compiled.c is very simple, I believe there are no issues related to libraries used.

Try to use something like gdb (possibly on the host with gdbserver on the target) to see what instruction is failing. — Chris Stratton, May 29 '14 at 02:18
Since my target machine does not have gdbserver, it is not possible to use GDB. (To install gdbserver on Target, I need to cross-compile binary for Target machine, meaning that my problem is solved) — freddy, May 29 '14 at 02:25
Can you use the cross objdump on both the working and non-working binaries to see if the instructions seem reasonable? Your system vendor or distributor should be able to point you to a working toolchain for the target though, and sources of utility programs there should include build flags. — Chris Stratton, May 29 '14 at 02:59
Could you post some more info? Perhaps the output of `/proc/cpuinfo` and `/proc/version` on the target? And have you tried first to cross compile the most simplest kind of "hello world" program to rule out issues with libraries? (For example) And if possible, the gcc command line used to do the compile and/or link? — 6EQUJ5, May 29 '14 at 03:17
@ChrisStratton I can objdump for the both binaries, but I could find unreasonable instructions (may be due to the lack of knowledge on MIPS instructions). — freddy, May 29 '14 at 04:46
@6EQUJ5 I updated what you requested. My binary program for Target is very simple (for testing purpose) as you said. — freddy, May 29 '14 at 04:47
Could you look at dmesg? It may contain more information about the illegal instruction. — Christophe Vu-Brugier, May 29 '14 at 05:56
@ChristopheVu-Brugier Sorry, there is no dmesg command line on my target machine. Could you let me know any alternatives? — freddy, May 29 '14 at 06:50

score 2 · Accepted Answer · answered May 29 '14 at 05:34

Various things could be causing this; one thing I can see from your post is that the ELF flags are different between your binary and their binary. The kernel is very old too, and you also haven't reported the version of gcc in your own buildroot.

I note that that chipset (BCM5354) is the same as used in the DLINK DIR-320 and others, that happen to be supported by OpenWRT. Assuming you are not already using OpenWRT as a buildroot, I would try again using OpenWRT.

Conversely, you could download the DLink provided GPL sources which should include a buildroot, from the DLINK support website and try that. Given the age of the kernel and gcc used to build your destination, and the newness of the trunk OpenWRT, you might want to try and find the oldest release of OpenWRT that supports that router, or try the DLINK build root first.

If you have a couple of computers you could try both buildroots in parallel...

If this doesn't fix it you'd need to be posting a lot more information

Could you let me know why the different flags can cause the problem? — freddy, May 29 '14 at 07:28
Any way of generalizing this answer? It resolves OP's problem, but doesn't give any way to actually solve it. — redbeam_, Aug 01 '15 at 22:33

score 2 · Answer 2 · answered May 30 '14 at 00:35

I compiled your hello world program using mipsel-linux-gcc (version 4.5.3). This is the output of mipsel-linux-objdump -d -S when ran on your hello world program. The program ran just fine on a Broadcom 7425 little endian machine I had on my desk.

004002a0 <main>:
  4002a0:       27bdffe0        addiu   sp,sp,-32
  4002a4:       afbf001c        sw      ra,28(sp)
  4002a8:       afbe0018        sw      s8,24(sp)
  4002ac:       03a0f021        move    s8,sp
  4002b0:       3c020040        lui     v0,0x40
  4002b4:       24444490        addiu   a0,v0,17552
  4002b8:       0c1000b8        jal     4002e0 <puts>
  4002bc:       00000000        nop
  4002c0:       00001021        move    v0,zero
  4002c4:       03c0e821        move    sp,s8
  4002c8:       8fbf001c        lw      ra,28(sp)
  4002cc:       8fbe0018        lw      s8,24(sp)
  4002d0:       27bd0020        addiu   sp,sp,32
  4002d4:       03e00008        jr      ra
  4002d8:       00000000        nop
  4002dc:       00000000        nop

Your dis-assembled compiler output should look remarkably similar to this. If it doesn't please update your question.

Thank you!. I've found "bal" instruction in the main function. Is it valid one for mips1 instruction set? — freddy, May 30 '14 at 07:53
I can't find any documentation on the MIPS 1 ISA, so I can't answer that question. You could try patching the binary with a `JAL` instead a `BAL` to see if that makes the problem go away. — markgz, May 30 '14 at 21:41

MIPS cross compilation error: illegal instruction

UPDATE:

2 Answers2