Client Bluetooth connection with 32feet.NET fails all the time

Question

I'm trying to get a Bluetooth socket connection up and running but for some reason my client will not connect.

More precisely I get an exception when I try to connect to the stream: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

All examples I found online didn't really solve my problem and I'm currently not really sure where the problem comes from. The scanning and pairing works fine - I see that the Bluetooth device in question gets successfully paired.

I try to connect via first setting the Client and then call connect

Client Bluetooth name, address and pin are known:

    public bool SetClient(String clientName, String btAddress, String pin)
    {
        bool retVal = false;
        m_remoteBluetoothClient = new BluetoothDeviceInfo(BluetoothAddress.Parse(btAddress));
        m_localBluetoothClient.SetPin(pin);
        if (m_remoteBluetoothClient.Authenticated)
        {
            //m_localBluetoothClient.Authenticate = true;
            retVal = true;
        }
        else
        {
            if (BluetoothSecurity.PairRequest(m_remoteBluetoothClient.DeviceAddress, pin))
            {
                retVal = true;
            }
        }
        return retVal;
    }

Then an async connect:

    private void ClientConnectThread()
    {
        m_localBluetoothClient.BeginConnect(m_remoteBluetoothClient.DeviceAddress, BluetoothService.SerialPort, Connect, m_localBluetoothClient);
    }

    private void Connect(IAsyncResult result)
    {
        if (result.IsCompleted)
        {
            m_localBluetoothClient.EndConnect(result);
            mBtStream = m_localBluetoothClient.GetStream();
        }
    }

The locals m_localBluetoothEndpoint and m_localBluetoothClient are created like this although the Endpoint is more or less new (before I used BluetoothCLient without parameter):

        m_localBluetoothEndpoint = new BluetoothEndPoint(BluetoothRadio.PrimaryRadio.LocalAddress, BluetoothService.SerialPort);
        m_localBluetoothClient = new BluetoothClient(m_localBluetoothEndpoint);

I also tried to work with a Listener in case the remote devices wants to connect but the callback gets never called:

    public void SetupListener()
    {
        var listener = new BluetoothListener(BluetoothService.SerialPort);
        listener.Start();
        listener.BeginAcceptBluetoothClient(this.BluetoothListenerAcceptClientCallbackTwo, listener);

    }

Can anyone tell me if there is anything wrong with my connection approach above and how I can figure out why I get the exception mentioned above?

The exception gets thrown here:

    m_localBluetoothClient.EndConnect(result);

A thing I also don't understand is that the SupportedServices call to the remoteCLient returned 0 guids - so the device did not list any Bluetooth services.

    m_remoteBluetoothClient.InstalledServices()

Thank you