Visual Studio 2010 hijacks Interop Callbacks and/or hides the true window handle

Question

I am working with a vendor provided non-managed DLL to interface with some custom hardware. Unfortunately I cannot post the vendor code, but hopefully the sanitized code below will provide enough information as to the cause of the problem.

The vendor SDK requires registering a callback function for hardware event notification. I have a utility from the vendor to confirm this happens successfully. The hardware sends custom WIN32 messages to the window, and the SDK attaches itself to the message loop and invokes the callback function whenever it receives one of these custom messages.

When the app is run directly, everything works. When the app is run inside of Visual Studio, registration succeeds but events are not fired. This is true running with or without debugging (F5 / Crlt+F5). I have also tried disabling the Visual Studio Hosting process in the project properties, and it has made no difference.

In a C# class library, I have wrapped the vendor SDK with the following code:

public class Basic {

    private delegate void Callback(UInt32 Event, Byte status, UInt16 type, UInt16 devno);
    private Callback mInstance;

    public delegate void EventHandler(String message);
    public event EventHandler HardwareEvent;

    [DllImport("VendorSDK.dll")]
    private static extern Int32 Register_Callback(Callback shuttleproc, UInt16 type, UInt16 devno);

    [DllImport("VendorSDK.dll")]
    private static extern Int32 Unregister_Callback(UInt16 type, UInt16 devno);

    public Basic() {
        mInstance = new Callback(CallbackHandler);

        if (Register_Callback(mInstance, 1, 1) != 0)
            throw new Exception("Error registereing");
    }

    ~Basic() {
        Unregister_Callback(1, 1);
    }

    public void CallbackHandler(UInt32 Event, Byte status, UInt16 type, UInt16 devno) {
        if (HardwareEvent != null)
            HardwareEvent(String.Format("Message: {0}, {1}, {2}, {3}", Event, status, type, devno));
    }
}

Then, in a WPF app I use this wrapper like so:

public partial class MainWindow : Window {
    Basic hardware;

    public MainWindow() {
        InitializeComponent();
    }

    protected override void OnSourceInitialized(EventArgs e) {
        base.OnSourceInitialized(e);
        hardware = new Basic();
        hardware.HardwareEvent += new Basic.EventHandler(HardwareEvent);
    }

    void HardwareEvent(string message) {
        Messages.Dispatcher.Invoke(new Action(() => { Messages.Items.Insert(0, message); }));
    }
}

The vendor SDK provided an unmanaged example in C++ that has the same issue as my managed wrapped. The app works fine outside of Visual Studio, but launched from inside of Visual Studio messages are not received.

I believe this is caused by Visual Studio either getting in the way of the callback, or more likely when the unmanaged vendor DLL access the current window handle (HWND) the wrong value is returned. I am not certain of this however.

This is unguessable, VS doesn't do things like this. Contact the vendor for support. — Hans Passant, Jan 05 '11 at 21:08
Does the API expose the HWND it has linked itself into? If so, I would compare that to data from something like Process Explorer (sysinternals tools). IIRC, there's also a sysinternals tool to monitor the messages getting dispatched... if it's wiring up correctly, you should be able to see the vendor messages show up. Another idea: Run outside of VS and attach? — Ben Von Handorf, Jan 05 '11 at 21:12
@Hans Passant - this is repeatable / run outside of visual studio it works, inside of visual studio it doesn't. The only change is using visual studio. — ViNull, Jan 05 '11 at 21:34
@Ben Von Handorf - there isn't a way to expose the hwnd the API has linked to. Running outside of VS and attaching works however (doesn't break the messages). Will have to search out the sysinternals tool - i thought stuff like that shipped with VS/Win SDK but it's been a while. — ViNull, Jan 05 '11 at 21:40
I think this was the right tool: http://technet.microsoft.com/en-us/sysinternals/bb896653 Edit: I was incorrect. I was thinking of Spy++, you are correct. — Ben Von Handorf, Jan 05 '11 at 21:51

score 0 · Answer 1 · answered Jan 12 '11 at 21:32

The cause appears to be related to Vista and permissions. By running Visual Studio as Administrator and running the vendors driver util as Administrator the window received messages. Any mismatch, even VS as user and driver util as Administrator, didn't work nor did running VS as user and the driver util as User.

This problem isn't happening on a Windows 7 system.

Thanks to Ben Von Handorf - the suggestion to attach the debugger led me down the patch to find out there is a permissions issue.

Visual Studio 2010 hijacks Interop Callbacks and/or hides the true window handle

1 Answers1