SIGSEGV when entering a function

Question

What can cause a segmentation fault when just entering a function?

The function entered looks like:

21:  void eesu3(Matrix & iQ)
22:  {

where Matrix is a struct. When running with GDB the backtrace produces:

(gdb) backtrace 
#0  eesu3 (iQ=...) at /home/.../eesu3.cc:22
#1  ...

GDB does not say what iQ is. The ... are literally there. What could cause this?

GCC: (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3

Program built with -O3 -g

The caller goes like:

Matrix q;
// do some stuff with q
eesu3(q);

Nothing special here

I reran the program with valgrind:

valgrind --tool=memcheck --leak-check=yes --show-reachable=yes --num-callers=20 --track-fds=yes <prgname>

Output:

==2240== Warning: client switching stacks?  SP change: 0x7fef7ef68 --> 0x7fe5e3000
==2240==          to suppress, use: --max-stackframe=10076008 or greater
==2240== Invalid write of size 8
==2240==    at 0x14C765B: eesu3( Matrix &) (eesu3.cc:22)
...
==2240==  Address 0x7fe5e3fd8 is on thread 1's stack
==2240== 
==2240== Can't extend stack to 0x7fe5e2420 during signal delivery for thread 1:
==2240==   no stack segment
==2240== 
==2240== Process terminating with default action of signal 11 (SIGSEGV)
==2240==  Access not within mapped region at address 0x7FE5E2420
==2240==    at 0x14C765B: eesu3( Matrix&) (eesu3.cc:22)
==2240==  If you believe this happened as a result of a stack
==2240==  overflow in your program's main thread (unlikely but
==2240==  possible), you can try to increase the size of the
==2240==  main thread stack using the --main-stacksize= flag.
==2240==  The main thread stack size used in this run was 8388608.

Looks like its a corrupted stack.

    Dump of assembler code for function eesu3( Matrix & ):
   0x00000000014c7640 <+0>: push   %rbp
   0x00000000014c7641 <+1>: mov    %rsp,%rbp
   0x00000000014c7644 <+4>: push   %r15
   0x00000000014c7646 <+6>: push   %r14
   0x00000000014c7648 <+8>: push   %r13
   0x00000000014c764a <+10>:    push   %r12
   0x00000000014c764c <+12>:    push   %rbx
   0x00000000014c764d <+13>:    and    $0xfffffffffffff000,%rsp
   0x00000000014c7654 <+20>:    sub    $0x99b000,%rsp
=> 0x00000000014c765b <+27>:    mov    %rdi,0xfd8(%rsp)

Okay, to make it clear: Matrix's data lives on the heap. It basically holds a pointer to the data. The struct is small, 32 bytes. (just checked)

Now, I rebuilt the program with different optimization options:

-O0: the error does not show.

-O1: the error does show.

-O3: the error does show.

--update

-O3 -fno-inline -fno-inline-functions: the error does not show.

That explains it. Too many inlines into the function led to excessive stack usage.

The problem was due to a stack overflow

To look at your variables, etc. do not optimize. Compile with `-O0 -g` — RageD, May 08 '12 at 13:18
Stack corruption is an interesting possibility. The error only shows on 1 machine. On all others I tried so far the program runs fine. This makes me think that the code is likely to be okay. Thing is i reinstalled Linux completely on that particular machine. And the error still shows. — ritter, May 08 '12 at 13:24
The error is absolutely reproducible. Always at that particular position — ritter, May 08 '12 at 13:25
Have you tried running it with `valgrind`? Curious if it shows any issues. — FatalError, May 08 '12 at 13:33
Are you dereferencing a pointer when you call `eesu3()`? (I.e. `eesu3(*pQ);`) — Kristof Provost, May 08 '12 at 13:46
@Frank: It looks like your function requires a huge stack frame, check the function local variables — Hasturkun, May 08 '12 at 14:28
From `--max-stackframe=10076008` I'd guess you have a stack overflow, because of a 10M matrix on the stack. — ugoren, May 08 '12 at 14:30
@Hasturkun The function has about 30 local instances of `Matrix`. Times 32 bytes, should be okay. Don't you think? — ritter, May 08 '12 at 14:41

Employed Russian · Accepted Answer · 2012-05-08T15:21:05.640

What can cause a segmentation fault when just entering a function?

The most frequent cause is stack exhaustion. Do (gdb) disas at crash point. If the instruction that crashed is the first read or write to a stack location after %rsp has been decremented, then stack exhaustion is almost definitely the cause.

Solution usually involves creating threads with larger stacks, moving some large variables from stack to heap, or both.

Another possible cause: if Matrix contains very large array, you can't put it on stack: the kernel will not extend stack beyond current by more than 128K (or so, I don't remember exact value). If Matrix is bigger than that limit, you can't put it on stack.

Update:

   0x00000000014c7654 <+20>:    sub    $0x99b000,%rsp
=> 0x00000000014c765b <+27>:    mov    %rdi,0xfd8(%rsp)

This disassembly confirms the diagnosis.

In addition, you are reserving 0x99b000 bytes on stack (that's almost 10MB). There must be some humongous objects you are trying to locate on stack in the eesu3 routine. Don't do that.

What do you mean by "the kernel will not extend stack beyond current by more than"

When you extend stack (decrement %rsp) by e.g. 1MB, and then try to touch that stack location, the memory will not be accessible (the kernel grows stack on-demand). This will generate a hardware trap, and transfer control to the kernel. When the kernel decides what to do, it looks at

Current %rsp
Meemory location that the application tried to access
Stack limit for the current thread

If faulting address is below current %rsp, but within 128K (or some other constant of similar magnitude), the kernel simply extends the stack (provided such extension will not exceed the stack limit).

If the faulting address is more than 128K below current %rsp (as appears to be the case here), you get SIGSEGV.

This all works nicely for most programs: even if they use a lot of stack in a recursive procedure, they usually extend stack in small chunks. But an equivalent program that tried to reserve all that stack in a single routine would have crashed.

Anyway, do (gdb) info locals at crash point, and see what locals might be requiring 10MB of stack. Then move them to heap.

Update 2:

No locals

Ah, the program has probably not made it far enough into eesu3 for there to be locals.

when building with -O0 the error disappears. GCC bug?

It could be a GCC bug, but more likely it's just that GCC is inlining a lot of other routines into eesu3, and each of the inlined routines needs its own N KBs of stack. Does the problem disappear if you build the source containing eesu3 with -fno-inline ?

Unfortunately, triage of such behavior and figuring out appropriate workarounds, or fixing GCC, requires compiler expertise. You could start by compiling with -fdump-tree-all and looking at generated <source>.*t.* files. These contain textual dumps of GCC internal representation at various stages of the compilation process. You may be able to understand enough of it to make further progress.

Payload of Matrix lives already on the heap. It really just owns basically a pointer to the data. Its not the size of struct Matrix (which is 8 or 16 bytes). See above output of `disas` — ritter, May 08 '12 at 14:20
@Employed Russian, What do you mean by "the kernel will not extend stack beyond current by more than" could you write more verbose please? — azat, May 08 '12 at 14:25
`info locals`: No locals. That's odd. There are 30 local vars defined, each 32bytes. Where have they gone? Possible effect of `-O3`? As in updated question when building with `-O0` the error disappears. GCC bug? — ritter, May 08 '12 at 14:58
@Frank: This might be caused by inlining, although GCC should be limiting the frame size. You could try using `-fno-inline`, `-fno-inline-functions`, etc to see if that's the issue — Hasturkun, May 08 '12 at 15:20
@Hasturkun Disabling the inline facilities solved it. I can still compile with `-O3`. Hope the performance does not degrade. — ritter, May 08 '12 at 16:44
What if it returns "No function contains program counter for selected frame." for `(gdb) disas`? — The Student, Nov 28 '13 at 16:52

score 5 · Answer 2 · answered May 08 '12 at 15:17

It's a stack overflow.

eesu3 tries to allocate something very large on the stack, which can be seen in its assembly code:

sub    $0x99b000,%rsp

This means more than 10MB of stack space are consumed.

The problem can be in eesu3 or an a function it calls, and the compiler chooses to inline.

My guess is that the problem is in a function eesu3 call, but not in the case you test (a debugging function?)
I guess this because it doesn't happen without optimization - with optimization, the function is inlined into eesu3, so eesu3 uses lots of stack. Without it, the function is not inline, so you'll get a problem only when it's actually called.

Yes, its likely to be a stack overflow. I hope its okay for you if @Emploed Russian gets the point. He was the first to point that out — ritter, May 08 '12 at 16:43
The point is there's no point in pointing the point out again. — hochl, May 09 '12 at 10:44

score 0 · Answer 3 · answered May 08 '12 at 13:42

0

If its a Matrix, check the indices you are trying to access. Maybe you are accessing elements that go beyond the dimensions of the Matrix object?

answered May 08 '12 at 13:42

tmaric

5,347
4
42
75

score 0 · Answer 4 · answered Oct 11 '12 at 10:30

You probably have some variables initialized in function

void eesu3(Matrix & iQ)

Allthough the debugger might step through variable declarations, they are probably initialized with the start of the scope (that is your function). If you would declare a very large buffer like so:

char * buffer[268435456];

You could get a stack overflow. It might be better to allocate some memory like

void * pvBuffer = malloc(268435456);

Have you declared a large buffer? Which is too large to put on the stack? It could mean that different architectures result in different possible maximum sizes for buffers (64 bit and 32 bit OSes)? Different kernels? As you said that the program runs fine on one machine but not on the other.

SIGSEGV when entering a function

4 Answers4