PinvokeStackImbalance error (C#)

Question

I'm a bit of a newbie to Pinvoke.

I have a large project that calls makes a call to a DLL called vtmail.dll. It worked fine in .Net 3.5 but on upgrading to .NET 4.0 I am getting a PinvokeStackImbalance exception

The calling code is a bit like this:

VTMail vtMail = new VTMail();
ec = vtMail.SendMail("servername",
   false,
   "",
   "",
   "Subject",
   "Sender Name",
   "sender@mydomain.com",
   "",
   "sendto@theirdomain.com",
   "reply@mydomain.com,
   "Description",
   "foldername",
   false,
   25);

Checking the declaration in vtmail.dll is this:

public int SendMail(string mailServer, bool login, string userName, 
                    string password, string subject, string fromName, 
                    string fromAddress, string toName, string toAddress, 
                    string replyToAddress, string bodyText, string folderToSend, 
                    bool encrypt, long smtpPortNo);

I've searched around about this and it seems to be a known .Net 4.0 issue. Responses have always suggested adding CallingConvention=CallingConvention.Cdecl to the DllImport statement, but this particular call doesn't have a DllImport statement, the dll is just added as a reference to the project in Visual Studio and is thus available.

How can I fix this?

It's not a .net issue. It's an issue with your code. You have a defect in your code. Sadly you did not show enough of the code. You managed to remove all the important code. We need the full C++ and C# signatures, and a description of the parameter semantics. — David Heffernan, Sep 11 '15 at 08:53
You cannot add a reference to an unmanaged DLL, probably you have a reference to a .NET wrapper DLL that contains DllImport attribute that you are looking for. — Sergey Rybalkin, Sep 11 '15 at 08:59
P/Invoke only applies to unmanaged code. If you have a reference to the DLL added to your project, then it's a managed DLL. Please attach the stack trace from the exception to your question. For all we know, the exception could be a bug inside `vtmail.dll` — theB, Sep 11 '15 at 09:11
@theB Thanks - I'm not getting a stack trace... Just a message saying 'A call to PInvoke function 'VTMail!MyNamespace.VTMail+VTMail32Dll::VTMail' has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the PInvoke signature match the target unmanaged signature.' Even wrapped in a `try... catch` it's not going into the `catch`, just dying instantly on that line. — komodosp, Sep 11 '15 at 09:57

score 5 · Accepted Answer · answered Sep 11 '15 at 09:24

    public int SendMail(..., long smtpPortNo);

It is excessively unlikely that the last argument is correct. Makes no sense at all to store a value that must be less than 65536 in a 64-bit long argument. It surely should be int instead. A classic Visual Basic accident, versions before VB.NET used Long to declare a 32-bit integer.

Why this didn't generate the diagnostic in 3.5 is hard to guess, it certainly should. Maybe you just never ran it with a debugger attached with the MDA enabled. Stack corruption caused by getting the declaration wrong has a habit of healing itself so it doesn't necessarily have to bomb at runtime. Still, a very, very unhealthy problem that can cause pretty random code execution.

Three basic things you can do about it. On top of the list with bells on is to use a telephone, you want the author of this library to fix his bug. Next is to try to flip the ignore bit and turn the MDA off: Debug > Exceptions > Managed Debugging Assistants, untick "PInvokeStackImbalance". And I probably ought to mention the new 4.0 config file option, <NetFx40_PInvokeStackResilience>, hesitantly.

But, really, that telephone must be your real fix.

Went with the second suggestion which worked... Had tried enabling `NetFx40_PInvokeStackResilience` before to no effect, and I probably won't be able to contact the author of the DLL any time soon... — komodosp, Sep 11 '15 at 10:29

PinvokeStackImbalance error (C#)

1 Answers1