0

In this case I do have the source for both the application and the dll.

When both are compiled without /NXCOMPAT, they work together fine. But when I compile both with /NXCOMPAT, I get a segfault deep in kernelspace.

If I compile the dll with /NXCOMPAT, and compile the executable without, it also works fine. (not surprising I suppose, since the DEP settings for the executable get forced on the loaded dll.)

I have previously seen a segfault in MainCRTStartup (Note: not the dll version), after enabling DEP, which was caused by another linker option. However, in this case that other linker option is NOT set, so I know that's not the answer.

Anyone have an idea where I should look for the cause?

Edit: Further strangeness. I've been running this in the debugger in VS 2008 the whole time, but when I tried running it without the debugger attached, the segfault disappears. I find this a very unsatisfactory solution, as I still don't know why it's been doing this.

Edit the 2nd: Also segfaults running in the debugger in VS 2013 Express.

Chris O
  • 5,017
  • 3
  • 35
  • 42
jbcarpen
  • 307
  • 2
  • 14

1 Answers1

1

Lacking code, we have to guess by the symptoms. My crystal ball says you're doing things inside LoadLibrary (i.e. inside DllMainCRTStartup) which are banned. And there is a very long list of things which are banned in LoadLibrary, including loading any other DLL.

Note that your global objects are created from DllMainCRTStartup and therefore have to respect the LoadLibrary rules as well.

MSalters
  • 173,980
  • 10
  • 155
  • 350
  • If I were doing something banned in DllMainCRTStartup, I'd expect a crash even without DEP active. And even if it was something banned by DEP, why would _that_ only crash _with a debugger attached?!?_ That's the part that has me really confused. – jbcarpen Jan 07 '15 at 18:25
  • @jbcarpen: That's the classical problem with Undefined Behavior. If the failure mode would have been consistent, it probably would have been easy to detect and specifically report the problem. – MSalters Jan 08 '15 at 07:42