1

I am building a dll for windows, using a Makefile, using cl.exe. I am using VS2015.. this dll uses CNG (bcrypt) for encryption operations, and bcryptr is loaded dynamically using loadlibrary call.

When i build with /Od option to disable optimization, i have no issues with any functionality. but if i use any optimization option /O1, /O2, /Ox, i see the strangest thing happen.. once i retrieve the address for a bcrypt function, such as BCryptGetFipsAlgorithmMode, using GetProcAddress, and then i make the call to that function ptr, the call stack goes away. This results in exception when the calling function tries to return.. it looks almost like when one calls a callback doesn't have the CALLBACK prefix, but i dont see the connection..

That bcrypt function's prototype looks like this:

NTSTATUS WINAPI BCryptGetFipsAlgorithmMode( __out BOOLEAN *pfEnabled)

and WINAPI seems to be defined:

define WINAPI __stdcall

Is there something I am missing? what does optimization have to do with this?

Any help would be appreciated.. Thank You!

Heres the code:

NTSTATUS GetFipsAlgorithmMode(BOOLEAN *pfEnabled )
{
   FARPROC pBCryptGetFipsAlgorithmMode = NULL;
   NTSTATUS (*_BCryptGetFipsAlgorithmMode)( __out  BOOLEAN *);
   NTSTATUS status = SPGC_ERR_LIBRARY_ADDRESS_LOOKUP_FAILURE;

   if(g_hBCRYPTDLL != NULL)
   {
      pBCryptGetFipsAlgorithmMode = GetProcAddress(g_hBCRYPTDLL, _T("BCryptGetFipsAlgorithmMode"));
      if(pBCryptGetFipsAlgorithmMode != NULL)
      {
         _BCryptGetFipsAlgorithmMode = (NTSTATUS (*)( __out  BOOLEAN *)) pBCryptGetFipsAlgorithmMode;

         status = _BCryptGetFipsAlgorithmMode(pfEnabled);
      }
   }
   return status;
}

step over the call to _BCryptGetFipsAlgorithmMode(), and the call stack basically gets cleared.

Community
  • 1
  • 1
scirdan
  • 59
  • 7
  • better show your code – RbMm Oct 25 '17 at 19:00
  • I surrounded this with #pragma optimize( "", off ) and #pragma optimize( "", on ), and the stack issue went away (exception went away).. its kind of a work around as this particular area being optimized is not important. – scirdan Oct 25 '17 at 20:57
  • 1
    are you use `__stdcall` calling convention as default ? if not - `NTSTATUS (*_BCryptGetFipsAlgorithmMode)( __out BOOLEAN *);` this is error - your forget `__stdcall` or `WINAPI`. must be `NTSTATUS (WINAPI *_BCryptGetFipsAlgorithmMode)( __out BOOLEAN *);` – RbMm Oct 25 '17 at 21:08
  • This was the problem! yeah this was a complete miss, i guess with no optimization i was getting lucky.. Looking good now, thanks for the help! – scirdan Oct 26 '17 at 04:23

0 Answers0