C# TcpListener and MySqlConnection stops accepting connections after a while

Question

I have an async socket server written in C#, running on a Lightsail server running Amazon Linux. It consists of a TcpListener that accepts connections, starts up a new thread to listen when someone connects, initiates an SSL connection, and then acts as a server for an online game.

This server works fine for about a day, until suddenly all networking stops working on the server. The crash takes anywhere from 22 hours to one week to occur. The symptoms are as follows:

Anyone already connected to the server will suddenly stop receiving/sending data. I can see in the logs that my inactivity checking code will eventually kick them for not sending heartbeat packets.
The server will also be unable to connect to its MySQL database (which is running on the same system, so it's unable to connect to localhost? I can still access it through PHPMyAdmin during this time).
It is, however, still able to write both to files and to console, as my logger is still able to write to both.

The code looks like everyone else's (I did try the changes suggested for this question, but it still crashed after ~24 hours). None of the errors get logged, so it looks like it never encounters an exception. No exceptions precede the crash, which is why I've been having problems figuring this one out.

For completeness, here is my main loop:

public void ListenLoop()
{
    TcpListener listener = new TcpListener(IPAddress.Any, 26000);
    listener.Start();

    while (true)
    {
        try
        {
            if (listener.Pending())
            {
                listener.BeginAcceptTcpClient(new AsyncCallback(AcceptConnection), listener);
                Logger.Write(Logger.Level.INFO, "continuing the main loop");
            }
            // Yield so we're not stuck in a busy-loop
            Thread.Sleep(5);
        }
        catch (Exception e)
        {
            Logger.Write(Logger.Level.ERROR, $"Error while waiting for listeners: {e.Message}\n{e.StackTrace}");
        }
    }
}

and here are the accept parts:

/// <summary>
/// Finish an async callback but spawn a new thread to handle it if necessary
/// </summary>
/// <param name="ar"></param>
private void AcceptConnection(IAsyncResult ar)
{
    if (ar.CompletedSynchronously)
    {
        // Force the accept logic to run async, to keep our listening
        // thread free.
        Action accept = () => AcceptCallback(ar);
        accept.BeginInvoke(accept.EndInvoke, null);
    } else
    {
        AcceptCallback(ar);
    }
}

private void AcceptCallback(IAsyncResult ar)
{
    try
    {
        TcpListener listener = (TcpListener) ar.AsyncState;
        TcpClient client = listener.EndAcceptTcpClient(ar);
        // If the SSL connection takes longer than 5s we have a problem, and should stop
        client.Client.ReceiveTimeout = 5000;

        // Attempt to get the IP address of the client we're connecting to
        IPEndPoint ipep = (IPEndPoint)client.Client.RemoteEndPoint;
        string ip = ipep.Address.ToString();
        Logger.Write(Logger.Level.INFO, $"Connection begun to {ip}");

        // Authenticate and begin communicating with the client
        SslStream stream = new SslStream(client.GetStream(), false);
        try
        {
            stream.AuthenticateAsServer(
                serverCertificate,
                enabledSslProtocols: System.Security.Authentication.SslProtocols.Tls12,
                clientCertificateRequired: false,
                checkCertificateRevocation: true
                );

            stream.ReadTimeout = 3600000;
            stream.WriteTimeout = 3600000;

            NetworkPlayer player = new NetworkPlayer();
            player.Name = ip;
            player.Connection.Stream = stream;
            player.Connection.Connected = true;
            player.Connection.Client = client;
            stream.BeginRead(player.Connection.Buffer, 0, 1024, new AsyncCallback(ReadCallback), player);
        }
        catch (Exception e)
        {
            Logger.Write(Logger.Level.ERROR, $"Error while starting the connection to {ip}: {e.Message}");
            // The following code just calls stream.Close(); and client.Close(); but sends exceptions to my logger.
            CloseConnectionSafely(client, stream);
        }
    }
    catch (Exception e)
    {
        Logger.Write(Logger.Level.ERROR, $"Error while starting a connection to an unknown user: {e.Message}");
    }
}

Taking a guess. Does issue happen around midnight? There may be issues with the timeouts when going from one day to next. — jdweng, Jul 26 '22 at 01:37
@jdweng It happens at a random point in time in the day, unfortunately, with no specific time that it happens. — OrdiNeu, Jul 26 '22 at 01:50
I think your accept callback method has issues. See the MSDN example for socket which should be the same as your code. The TCP listener inherits the socket Listener : https://learn.microsoft.com/en-us/dotnet/framework/network-programming/asynchronous-server-socket-example — jdweng, Jul 26 '22 at 08:57

score 1 · Answer 1 · answered Jul 26 '22 at 01:38

I'm guessing that your primary issue is that you are not disposing the stream and therefore you are getting socket exhaustion.

Apart from that I would advise you to move to fully async code using Task.

public async Task ListenLoop(CancellationToken cancel)  // use a cancellation token to shutdown the loop
{
    using (var TcpListener listener = new TcpListener(IPAddress.Any, 26000))
    {
        listener.Start();

        while (!cancel.IsCancellationRequested)
        {
            try
            {
                var client = await listener.AcceptTcpClientAsync(cancel);
                Task.Run(async () => await AcceptConnection(client, cancel));
                Logger.Write(Logger.Level.INFO, "continuing the main loop");
                // no need to yield due to async
            }
            catch (OperationCanceledException) { }
            catch (Exception e)
            {
                Logger.Write(Logger.Level.ERROR, $"Error while waiting for listeners: {e.Message}\n{e.StackTrace}");
            }
        }

        listener.Stop();
    }
}


private async Task AcceptConnection(TcpClient client, CancellationToken cancel)
{
    try
    {
        using (client)
        {
            // If the SSL connection takes longer than 5s we have a problem, and should stop
            client.Client.ReceiveTimeout = 5000;
            await AcceptConnectionImpl(client, cancel);
        }
    }
    catch (OperationCanceledException) { }
    catch (Exception e)
    {
        Logger.Write(Logger.Level.ERROR, $"Error while starting a connection to an unknown user: {e.Message}");
    }
}

private async Task AcceptConnectionImpl(TcpClient client, CancellationToken cancel)
{
    // Attempt to get the IP address of the client we're connecting to
    IPEndPoint ipep = client.Client.RemoteEndPoint;
    Logger.Write(Logger.Level.INFO, $"Connection begun to {ipep.Address}");

    // Authenticate and begin communicating with the client
    using (SslStream stream = new SslStream(client.GetStream(), false))
    {
        try
        {
            await stream.AuthenticateAsServerAsync(
                serverCertificate,
                enabledSslProtocols: System.Security.Authentication.SslProtocols.Tls12,
                clientCertificateRequired: false,
                checkCertificateRevocation: true
                );

            stream.ReadTimeout = 3600000;
            stream.WriteTimeout = 3600000;

            NetworkPlayer player = new NetworkPlayer();
            player.Name = ip;
            player.Connection.Stream = stream;
            player.Connection.Connected = true;
            player.Connection.Client = client;
            player.Cancellation = cancel;
            await player.YourReadLoopAsync();
        }
        catch (OperationCanceledException) { }
        catch (Exception e)
        {
            Logger.Write(Logger.Level.ERROR, $"Error while starting the connection to {ip}: {e.Message}");
            // The following code just calls stream.Close(); and client.Close(); but sends exceptions to my logger.
            CloseConnectionSafely(client, stream);
        }
    }
}

The function YourReadLoopAsync should read data from the stream using ReadAsync, or using classes like StreamReader which also has async functions.

You don't need to use CancellationToken, but it does make it easier to deal with shutting everything down cleanly. Make sure to catch OperationCanceledException on every try.

See also this link for further tips.

I'm not certain, but does `IPGlobalProperties.GetIPGlobalProperties().GetTcpIPv4Statistics().CurrentConnections` get me the number of sockets in use? At the time of crashing, it is only at 19. In every case where I think the client closes, I call `CloseConnectionSafely` which calls `try {stream.Close();} finally {client.Close();}`, so it should be getting closed properly. — OrdiNeu, Jul 26 '22 at 01:56
Hmm so it may not be that, more likely something else not getting closed, or perhaps a deadlock. There are certainly many objects that you are not disposing, irrespective of the number of connections these may leak also. You only seem to be calling that in a `catch`. Either way a `using` is far more straght-forward: it's basically `try { DoStuff; } finally { client.Close(); }`. So you should be doing this anyway. Moving to `async` may also prevent a deadlock. — Charlieface, Jul 26 '22 at 02:00

score 0 · Accepted Answer · answered Sep 29 '22 at 14:15

The solution I found after consulting some people more familiar with C# than me is that I was running into Thread Pool Exhaustion. Essentially, I had a bunch of other async tasks (not shown in the code in the question, as they didn't look like they could cause what I was seeing) that were stuck executing some extremely-long-IOs (talking to users that had either disconnected improperly or were behind very high latency), which prevented the async AcceptCallback in my post from being picked up by the Thread Pool. This had a myriad of other side-effects which I outlined in the question:

Creating a new connection to a MySQL database involves an async task behind-the-scenes, which was being starved out due to exhaustion.
Completing the EndAcceptTcpClient required my async task to run, which requires an available thread.
Tasks which did not involve the async keyword, such as Timer() bound tasks (like my logger I/O) were unaffected and could still run.

My solution involved reducing the number of synchronization steps elsewhere in my program, and restructuring any tasks that could take a long time to execute so that they didn't block threads. Thank you to everyone who looked/commented.

C# TcpListener and MySqlConnection stops accepting connections after a while

2 Answers2