0

I am looking for the most fool-safe way to pass a VB6 boolean variable to a function (written in C++, stdcall).

The C++ function will set the "bool" variable of a struct using this VB6 variable.

I have tried declaring it like this in C++:

extern "C" __declspec(dllexport) int SetParameter( BOOL bErrorView) 
{

    DLL_sSettings nSet;
    nSet.bErrorView =(bErrorView != FALSE);

    int ret = stSetParameter(sizeof(DLL_sSettings), nSet);
    return (ret);
}

stSetParameter is declared as

extern "C" int ST_COMDLL_API stSetParameter(int DataLen, DLL_sSettings Settings);

The DLL_sSetting is declared as 

typedef struct
{
    bool bErrorView;            // true:    Show 
                                // false:   Don't show
    (...)
} DLL_sSettings;

However, I am unable to get it to work.

I call it in VB6 using 

Private Declare Function SetParameter Lib "MyDLL.dll" Alias "_SetParameter@4" (ByVal bErrorView As Boolean) As Long

But it does not work as expected, I guess somewhere the VB6 Boolean gets lost or is being incorrectly converted.

I am currently using VB6 Boolean, C++ BOOL and C++ bool. I know that is not so nice, but I don't see any other way.

Does anybody spot something wrong in my code?

tmighty
  • 10,734
  • 21
  • 104
  • 218
  • It looks like the STDCALL function and the one VB6 is using are two different functions. Make `SetParameter` stdcall as well. – Ben Voigt Nov 09 '14 at 22:21
  • @BenVoigt My C++ DLL is set to calling convention __stdcall (/Gz) via its properties. Should I write "extern "C" int SetParameter"? Could you give me the exact spelling? And should I keep the "extern C"? – tmighty Nov 09 '14 at 22:31
  • `extern "C" __declspec(dllexport) int __stdcall SetParameter(BOOL bErrorView)` should be fine. And when using `/Gz` it should not be necessary to specify `__stdcall`, but it doesn't hurt anything so I would make it explicit. – Ben Voigt Nov 09 '14 at 22:34
  • @BenVoigt Ah yes... I am getting there. Passing a VB6 Boolean True variable already gives the correct results (while it actually did not do aynthing before I changed the declaration as you suggested), but passing a False does not do anything. That is why I am wondering if the (bErrorView != FALSE); is really correct. – tmighty Nov 09 '14 at 22:41
  • Try changing the VB6 argument to `ByVal Long`. If VB6 stores only one byte, and C++ reads four, then even though the first byte is zero, junk in the others could make the overall value be `!= FALSE`. – Ben Voigt Nov 09 '14 at 22:43
  • @BenVoigt Unfortunately this did not help. I am still getting the same results, even with Longs instead of Booleans. – tmighty Nov 09 '14 at 22:50
  • So it is treating `bErrorView != 0` as true no matter what you pass in? Have you used the debugger to put a breakpoint in the C++ code, and see what value `bErrorView` has? Does it look like a VB6 boolean value (0 or -1)? Like a WINAPI BOOL (0 or 1)? Like a memory address? – Ben Voigt Nov 09 '14 at 22:53
  • @BenVoigt True = -1255 False = -655360 – tmighty Nov 09 '14 at 22:57
  • Hex is much better for this. The false one (`0xFFF60000`) appears to have zero in the low bytes. Umm, is that -1255 or -1 (255)? And likewise -65536 (0)? This looks somewhat like you used `Integer` (2 bytes) instead of `Long` (4 bytes). – Ben Voigt Nov 09 '14 at 23:00
  • Oh, I don't know. I used a brutal approach to get these values, namely std::stringstream ss; ss << bErrorView; MessageBox(0, ss.str().c_str(), "MSG", MB_OK); I have tried it again now, and now I get -1 for True, and -65536 for False. Does this help? I tried to debug it by attaching the VSIDE to it, but I was for some reason unable. Still too unexperienced. – tmighty Nov 09 '14 at 23:03
  • Ahh, you didn't put a space or any separator in between. So yes, it's almost certainly `-1,255` and `-65536,0`. In that case, try `nSet.bErrorView = (bErrorView & 0xFF) != 0;` meaning just look at the lowest byte. – Ben Voigt Nov 09 '14 at 23:05
  • @BenVoigt I have tried that with the same results. I have now tried to enforce it manually by just setting the member manually in the DLL like so: set.bErrorView = false; set.bErrorView = FALSE; set.bErrorView = 0; set.bErrorView = -1; No change whatsoever. – tmighty Nov 09 '14 at 23:21
  • So you're saying that the problem is between the C++ wrapper and the DLL it wraps, and the VB6 code is no longer involved? Are you sure the `sizeof(DLL_sSettings)` value is coming out right? Does `ret` indicate an error in calling the other DLL? – Ben Voigt Nov 10 '14 at 00:48
  • Note that all the rest of your structure is left uninitialized. The other members could easily have illegal values and cause the call to fail. How does `stSetParameter` know that `bErrorView` is the value you care about, as opposed to all the other options in the same structure? – Ben Voigt Nov 10 '14 at 00:50
  • @BenVoigt I got it!!! I only missed to include your changes in all of my functions. Can you make your comment "extern "C" __declspec(dllexport) int __stdcall" the reply / answer? Thank you! – tmighty Nov 10 '14 at 01:35

2 Answers2

2

BOOL is type definition for int.

it declared in windef.h as follows:

typedef int                 BOOL;

#ifndef FALSE
#define FALSE               0
#endif

#ifndef TRUE
#define TRUE                1
#endif

bool is C++ type, which can't be used in function prototype if you declare function with extern "C".

so VB should treat BOOL as Long (32 bit integer), not as Bolean. 0 means false otherwise (usually 1) it is true.

SHR
  • 7,940
  • 9
  • 38
  • 57
  • The struct is fine, VB6 isn't touching it. The sole concern is matching between the function signature in C++ and in VB. – Ben Voigt Nov 09 '14 at 22:18
  • Ok, I removed the last line, the base answer is to use `int` in the interface, and check if it is 0 or not. – SHR Nov 09 '14 at 22:33
  • 2
    VB6 uses `-1` for True, but because the C++ code uses `!= FALSE`, it should accept `1`, `-1`, and anything that isn't zero. And VB should use `Long` (32-bits) for WinAPI `BOOL`. – Ben Voigt Nov 09 '14 at 22:36
  • It is integer, in VB only Boolean value is permitted in condition, and AFAIK, it does not allow something like `if(1) then...`, anyway should check `if (var=0)/(var!=0)` not `if (var=false/true)`. and for long in VB you are right. – SHR Nov 09 '14 at 22:39
  • You're really mixing up VB6 and C++ with that last comment – Ben Voigt Nov 09 '14 at 22:43
2

VB6 uses the StdCall calling convention by default (cdecl convention is supported if you create a type library with a module section describing your imports, instead of using Declare Function). And C++ supports a whole host of calling conventions: stdcall, fastcall, cdecl, thiscall.

It is important to note that calling functions using stdcall in another library is not enough to change your functions to stdcall. You can use a command-line switch to the compiler, but the most robust is to include the __stdcall keyword in your source code. It gets applied to the name of the function, like so:

int __stdcall functionname(int args);

Since you will also want to export those functions for VB6 to find them, you'll want extern "C" to reduce name mangling, and __declspec(dllexport) to place them in the exports table.

It's pretty common to use a macro to do all of the above at once. It looks like the library you are wrapping does this with ST_COMDLL_API. Inside the library, that will expand to __declspec(dllexport) __stdcall. For consumers, it will use dllimport instead.

Any time you are defining an API to be used across different compilers (or even different languages), it's a good idea to be very explicit about calling convention and structure packing (#pragma pack). Otherwise you are at the mercy of options specified in the project file and other headers files. Even if you are happy with the defaults the compiler uses, your public headers should be explicit, because eventually someone will try to use two libraries in the same program, and the other library may demand changes to the compile options.

Ben Voigt
  • 277,958
  • 43
  • 419
  • 720