2

So, here I've got some complex client-server application and some part of it opens port for listening and further network interactions. QTcpServer + QTcpSocket are used (the details will be described below).

This "network layer" of the application worked fine everywhere, except Windows 8.1 Pro. Just don't ask, why our client deсided, that it could be used as a server... It occured, that incorrect restarting of the process, which opened TCP connection on a specific port for listening causes sometimes this port to be unsuitable for any following attempts to bind it. It looks like some sort of magic, but the behavior occured as follows:

  • Port 8554 was listened ("conection -- established") by a process
  • Process crashes or killed somehow while having clients connected via this socket
  • Process restarts and attempts to listen on the port again. It fails with "already in use".
  • I stop the server and try to check the port via netstat -an. It's free.
  • I wait for some time and try to check the port via powershell, like:

    $Listener = [System.Net.Sockets.TcpListener]8554
    $Listener.Start()
    

    No, this leads to the same error like "already in use".

  • I can bind socket to the other port, powershell snippet also works on them. But emergency restart of our server "breaks" them as well, scenario is the same.
  • Once the port is "broken", windows restart is the only cure to it.
  • Binding to "any address", i.e. 0.0.0.0:8554 behaves as described. Binding to exact IP, like 10.11.12.123:8554 is better, checked it while trying to bind FileZilla on the "broken port".

Now goes the coding problem. Providing exact IP for binding looks like a bad idea, a least in our architecture, so I decided to use SO_REUSEADDR on Windows. But looks like I have to set this option before bind/listen calls, which leads to significant QTcpServer usage customizing. Keep in mind that the application is crossplatform (WinSock + sys/socket + some #defines if something other than Qt methods are to be used...). Aaaand there goes my favourite legacy version of QTcpServer customization, take a look:

QTcpServer2.h

#pragma once

#include <QTcpServer>
#include <QMutex>

class QTcpServer2 : public QTcpServer
{
  QMutex         mConnectionMutex;
  QList<qintptr> mSocketDescriptors;

private:
  virtual void incomingConnection(qintptr socketDescriptor) override;

public:
  bool TakeIncomingSocketDescription(qintptr& socketDescriptor);

public:
  QTcpServer2();
};

QTcpServer2.cpp

#include <QMutexLocker>

#include "QTcpServer2.h"

void QTcpServer2::incomingConnection(qintptr socketDescriptor)
{
  QMutexLocker lock(&mConnectionMutex);
  mSocketDescriptors.append(socketDescriptor);
}

bool QTcpServer2::TakeIncomingSocketDescription(qintptr& socketDescriptor)
{
  QMutexLocker lock(&mConnectionMutex);
  if (mSocketDescriptors.empty()) {
    return false;
  }
  socketDescriptor = mSocketDescriptors.takeFirst();
  return true;
}

QTcpServer2::QTcpServer2()
{ }

Usage:

bool NetServer::DoInitConnection()
{
  mNetServer = QSharedPointer<QTcpServer2>(new QTcpServer2);
  if (!mNetServer->listen(QHostAddress::AnyIPv4, mPort)) {
    Log.Fatal(QString("Listen port fail (port: %1)").arg(mPort), true);
    return false;
  }
  return true;
}

Sigh... Yeah, I know about pending connections mechanism, about signal-slot idea, but that's out code for now. TakeIncomingSocketDescription is used to pass socket descriptor somewhere. And threaded access, yes. Anyway, this piece of... code needs refactoring and I badly need your piece of advice: what's the apropriate way to customize socket binding here? Assume that subclassing stays, list of descriptors is also something not-so-easy to get rid of.

Qt Sources showed me the following:

/*! \internal
*/
void QTcpServerPrivate::configureCreatedSocket()
{
#if defined(Q_OS_UNIX)
    // Under Unix, we want to be able to bind to the port, even if a socket on
    // the same address-port is in TIME_WAIT. Under Windows this is possible
    // anyway -- furthermore, the meaning of reusable on Windows is different:
    // it means that you can use the same address-port for multiple listening
    // sockets.
    // Don't abort though if we can't set that option. For example the socks
    // engine doesn't support that option, but that shouldn't prevent us from
    // trying to bind/listen.
    socketEngine->setOption(QAbstractSocketEngine::AddressReusable, 1);
#endif
}

That's exactly what I want, but there is no access to these internals from the Qt classes interface. QTcpServer performs bind call implicitly, so I cannot pass QAbstractSocket::ReuseAddressHint there.

Possibly, there is some neat solution, possibly not. I would be grateful for any ideas.

MasterAler
  • 1,614
  • 3
  • 23
  • 35
  • 1
    Does Win 8.1 Pro implement a `POSIX.1-2001` layer? If so a call to `::setsockopt(mNetServer.socketDescriptor(), SOL_SOCKET, SO_REUSEADDR, ...)` should do the trick and should be portable across most OSs. – G.M. Jan 16 '19 at 12:41
  • @G.M., Maybe I'm wrong, but socketDescriptor is set either in `listen()` implicitly or in `setSocketDescriptor()` explicitly, where it has to be in a "listen" state. Do you mean I'll have to implement everything -- configuring, binding & listening of the socket myself? оО – MasterAler Jan 16 '19 at 12:51

1 Answers1

0

Took me a while to consider all the possible solutions, though there were not many of them at all.

  1. Here I found something looking much like my problem. Made an adaptation of the code, provided there... Okay, just took most of it from that answer =)
  2. @G.M.'s comment pushed me to the idea that socket api in this case should differ only in headers

Manuals told me that SO_LINGER flag is often a result of bad architecture (client should close the connection in most cases, not server), after several experiments it occured this flag wasn't of much use in my case.

Binding to the exact IP lead to several complexities, more of a crutch than a solution IMHO.

So, after minor adaptations the socket listening class became something like that:

QTcpServer2.h

#pragma once

#include <QTcpServer>
#include <QMutex>

class QTcpServer2 : public QTcpServer
{
  QMutex            mConnectionMutex;
  QList<qintptr>    mSocketDescriptors;
  const int         mPort;
  bool              mConnectionSuccessful;

private:
  /*override */virtual void incomingConnection(qintptr socketDescriptor) Q_DECL_OVERRIDE;

public:
  bool TakeIncomingSocketDescription(qintptr& socketDescriptor);

  bool isConnectionSuccessfull() const
  { return mConnectionSuccessful; }

public:
  QTcpServer2(int port);
};

QTcpServer.cpp

#include "QTcpServer2.h"

#include <QMutexLocker>

#ifdef Q_OS_WIN
    #include <Windows.h>
    #pragma comment(lib, "ws2_32.lib")
#endif

#ifdef Q_OS_LINUX
    #include <sys/socket.h>
    #include <netinet/in.h>
    #include <netinet/tcp.h>
#endif

void QTcpServer2::incomingConnection(qintptr socketDescriptor)
{
    QMutexLocker lock(&mConnectionMutex);
    mSocketDescriptors.append(socketDescriptor);
}

bool QTcpServer2::TakeIncomingSocketDescription(qintptr& socketDescriptor)
{
    QMutexLocker lock(&mConnectionMutex);
    if (mSocketDescriptors.empty()) {
        return false;
    }
    socketDescriptor = mSocketDescriptors.takeFirst();
    return true;
}

QTcpServer2::QTcpServer2(int port)
    : QTcpServer()
    , mPort(port)
    , mConnectionSuccessful(false)
{
    // open server and listen on given port
    int sockfd = 0;
    struct sockaddr_in serv_addr;
    memset(&serv_addr, 0, sizeof(serv_addr));

    sockfd = ::socket(AF_INET, SOCK_STREAM, 0);
    if(sockfd < 0)
    {
        qDedug() << "QTcpServer2: socket couldn't be opened successfully!";
        return; //RET
    }

#ifdef Q_OS_WIN
    // Not required in Linux, won't make any good 
    int flag = 1;
    if(::setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, reinterpret_cast<char*>(&flag), sizeof(int)) < 0)
    {
        qDedug() << "QTcpServer2: Can't set SO_REUSEADDR";
        return; //RET
    }
#endif

    //set Address,IFace, Port...
    serv_addr.sin_family = AF_INET;
    serv_addr.sin_addr.s_addr = INADDR_ANY;
    serv_addr.sin_port = htons(static_cast<ushort>(mPort));


    if (::bind(sockfd, (struct sockaddr *) &serv_addr, sizeof(sockaddr_in)) < 0)
    {
        qDedug() << "QTcpServer2: can't bind socket" ;
        return; //RET
    }

    if(::listen(sockfd, SOMAXCONN) < 0)
    {
        qDedug() << "QTcpServer2: can't listen on port";
        return; //RET
    }

    //forward our descriptor with SO_REUSEPORT to QTcpServer member
    setSocketDescriptor(sockfd);
    mConnectionSuccessful = true;
    qDedug() << "QTcpServer2: socket success =)";
}

As long as the "magical problem with sockets" appeared on Windows 8.1 only, #define makes SO_REUSEADDR used on Windows only. On Linux Qt already sets this flag by itself (as shown in the question), so this small patch only improved Windows behavior to match the desired one, provided on Linux without problems.

Hopefully, nothing like that would ever be of need on other platforms.

MasterAler
  • 1,614
  • 3
  • 23
  • 35