Stack Smashing when returning pointer to array

Question

my function neldermead looks like this:

double * neldermead (double data[], double (*function)(int, double, double, double, double, double),
double ia1, double ia2, double ia3, double ia4, double ia5, double rad, int k) {

...

printf("8\n");
double * xout = malloc(5 * sizeof(double));

xout[0] = x[0][0];  
xout[1] = x[1][0]; 
xout[2] = x[2][0];  
xout[3] = x[3][0];  
xout[4] = x[4][0];

printf("10\n");  
return xout;
}

The function is called as such:

...
double * newX;
printf("11\n");  
newX = neldermead (data,  &function1, F1A1, F1A2, F1A3, F1A4, F1A5, 10, 10000);
printf("12\n");  
...

The output is as follows:

11
8
10
*** stack smashing detected ***: ./a.out terminated
Aborted (core dumped)

With -fno-stack-protector the code works fine.

Is there something wrong with how I returned the array or what?

////////////////////////////////////

I modified the code to do return by reference.

my function neldermead looks like this:

double * neldermead (double * xout, double data[], double (*function)(int, double, double, double, double, double),
double ia1, double ia2, double ia3, double ia4, double ia5, double rad, int k) {

...

printf("8\n");


xout[0] = x[0][0];  
xout[1] = x[1][0]; 
xout[2] = x[2][0];  
xout[3] = x[3][0];  
xout[4] = x[4][0];

printf("9\n");  

}

The function is called as such:

...
double newX[5];
printf("10\n");  
neldermead (newX, data,  &function1, F1A1, F1A2, F1A3, F1A4, F1A5, 10, 10000);
printf("11\n");  
...

The output is as follows:

10
8
9
*** stack smashing detected ***: ./a.out terminated
Aborted (core dumped)

Damn, didn't help.

coding style: usually, you'd want the function that malloc's memory to take ownership, but that's not the problem here, most likely. — Marcus Müller, Oct 09 '16 at 04:33
I don't see anything wrong in the code you posted, but there could be a problem elsewhere in the function. Could you post the complete code to neldermead please? — Antimony, Oct 09 '16 at 04:42
Your function takes 5 parameters in declaration, you are not passing them — Makketronix, Oct 09 '16 at 05:27
It would be best to put together a short complete example as the issue is likely in the code you didn't show. — Retired Ninja, Oct 09 '16 at 06:53
weirdly, using return by reference gives less accuracy than return by calloc. — BostonBrooks, Oct 09 '16 at 07:18

score 0 · Answer 1 · answered Oct 09 '16 at 05:30

0

The function

 double (*function)(int, double, double, double, double, double)

Takes parameters. When function returns, it cleans up the stack. It expects parameters to be have been pushed on the stack.

You call it with:

newX = neldermead (data,  &function1, F1A1, F1A2, F1A3, F1A4, F1A5, 10, 10000);

You are not passing parameters to the function. When it returns, it returns to wrong address.

answered Oct 09 '16 at 05:30

Makketronix

1,389
1
11
31

no, neldermead calls function1 for a variety of int values. – BostonBrooks Oct 09 '16 at 05:39

Stack Smashing when returning pointer to array

1 Answers1