2

What can cause MSVCRT system() function to always return -1 error code, even if the application is executed and exited successfully and returning 0 as its exit code?

Ive made tests with TDM-GCC-4.9.2 and FASM that only calls system() and printing the return code, and they all print -1 so its not my dev environment.

Also errno is 0 after the call to system() and GetLastError returns 18 There are no more files, which is odd.

In fact every application on my system that uses system() now always assumes it failed.

So this is a global problem with MSVCRT that i cant seem to reproduce on any other machine, some help would be appreciated.

EDIT: After some debugging, system calls _spawnve which in turn calls CreateProcessA and then WaitForSingleObject. After the executable terminates it calls GetExitCodeProcess and that is the one that returns -1 which then gets fed back up the chain.

EDIT 2: After some more testing, it seems system only returns -1 if the called process returns 0 otherwise it returns the correct value.

EDIT 3: Just to clarify, even though system returns -1, the called process is executed successfully.

EDIT 4: Here are my test sources.
The one that always return 0 success:

#include <stdio.h>

int main( int argc, char* argv[argc]) {
    printf("success\n");
    return 0;
}

The one that always fails:

#include <stdio.h>

int main( int argc, char* argv[argc]) {
    printf("failure\n");
    return 1;
}

And the one that calls it:

#include <stdlib.h>
#include <stdio.h>

int main( int argc, char* argv[argc]) {
    printf( "success == %d %d\n", system("test_success.exe", errno);
    printf( "failure == %d %d\n", system("test_fail.exe", errno);
    return 0;
}

The output of this is:

success
success == -1 0
failure
failure == 1 0

EDIT 5: Since system calls _spawnve calls CreateProcess i tried them all, and they all return -1 when calling cmd /c test_success but when calling cmd /c test_fail they work as expected. So this seems to be a deeper issue not directly related to system.

EDIT 6: After much faffing around i changed ComSpec to C:\Windows\winsxs\amd64_microsoft-windows-commandprompt_31bf3856ad364e35_6.1.7601.17514_none_e932cc2c30fc13b0\cmd.exe and everything works now! Which is a bit weird since im on an Intel Core 2 Duo, and its probably not the right thing to do here, but still im satisfied :p

grable
  • 21
  • 3
  • Have you tried stepping into the system() function with a debugger to see what is happening? Also, check the value of the ComSpec environment variable, and that `cmd.exe` itself hasn't been corrupted in some way. – Harry Johnston Apr 21 '15 at 00:39
  • @HarryJohnston: COMSPEC is "C:\Windows\System32\cmd.exe" and calling it recursively with /c works as expected. I will try debugging it though... – grable Apr 21 '15 at 00:47
  • did you try using GetLastError() and check for error code? – GingerJack Apr 21 '15 at 02:43
  • @GingerJack: GetLastError returns 18 "There are no more files" which just doesnt make any sense. – grable Apr 21 '15 at 07:17
  • If it returns -1 then you are supposed to use *errno* to obtain the error code. Do not use GetLastError(). Document the value of errno you see. – Hans Passant Apr 21 '15 at 08:06
  • Ignore GetLastError, MSDN documents that errno is set in case of an error and it does not say anything about GetLastError. It might just be that system does not change the last error and it is the one that was set before you called system. – Werner Henze Apr 21 '15 at 08:08
  • 1
    Please show your exact code. If it is `system("ls")` then your test is wrong, because ls does not work on Windows/cmd.exe. – Werner Henze Apr 21 '15 at 08:10
  • @HansPassant: I already did, and its 0. There are only 4 documented error codes for system, and none of them come as far as starting the executable. – grable Apr 21 '15 at 08:13
  • @WernerHenze: I used LS because it has a short name, and i have a win32 version that works just fine in windows and cmd.exe, LS is not the issue. But il revise the question. – grable Apr 21 '15 at 08:19
  • Hard to guess what kind of disease your system() suffers from. If you get a 0 for errno then there was no error. So just ignore it, I suppose. You *can* debug it, you've got the source of the CRT on your machine. Compile with /MT. – Hans Passant Apr 21 '15 at 08:20
  • @HansPassant: Yeah, im stumped too. I have no sources as i dont have visual studio, only mingw. I did step through it in OllyDebug though but didnt find anything that stood out. – grable Apr 21 '15 at 08:35
  • Works here with MSVC 2010. To narrow it down I would run cmd.exe and first `test_success.exe & echo %errorlevel%` and second `cmd /c test_success.exe & echo %errorlevel%`. – Werner Henze Apr 21 '15 at 09:17
  • @WernerHenze: When called manually in **cmd** they both work as expected, but i did some more tests using `_spawnve` and `CreateProcess and WaitForSingleObject and GetExitCodeProcess` directly and calling `cmd /c test_success` and they all fail with **-1** but `cmd /c test_fail` works. So it seems to be related to calling **cmd** for some reason. – grable Apr 21 '15 at 11:33

2 Answers2

0

Im adding an answer to myself because i consider this done.

Changing ComSpec from C:\Windows\System32\cmd.exe to C:\Windows\winsxs\amd64_microsoft-windows-commandprompt_31bf3856ad364e35_6.1.7601.17514_none_e932cc2c30fc13b0\cmd.exe solved all the issues i had with bogus exit codes.

grable
  • 21
  • 3
-1

here is what the manual for 'system()' has to say about the returned value

   The value returned is -1 on  error  (e.g.,  fork(2)  failed),  and  the
   return  status  of the command otherwise.  This latter return status is
   in the format specified in wait(2).  Thus, the exit code of the command
   will  be  WEXITSTATUS(status).   In case /bin/sh could not be executed,
   the exit status will be that of a command that does exit(127).

   If the value of command is NULL, system() returns nonzero if the  shell
   is available, and zero if not.

   system() does not affect the wait status of any other children.
user3629249
  • 16,402
  • 1
  • 16
  • 17