Why does my server's performance suffer after operating for some time?

Question

I have written server code which receives the data from servers continuously using multithreading (asynchronous) from multiple clients (more than 500). It reads data from all the clients and works properly in terms multithreading and receives all the message sent by clients.

What is the problem ?

The problem is when I keep on reading data from client after 1 and 2 hours my server gets slow. By that, I mean it keeps on increasing CPU usage (may be be because of unfreeing the memory…because here we leave every thing on garbage collector). When I see in CPU the memory rises from 14% to suddenly 80-100% after some time. But it still receives data from client (but some there is leak of data) but it still receives. And I am not able to determine the cause of the problem.

Could someone please let me know what could be the basic cause of it?

The important part of code is:

public void StartServer() 
{
    try 
    {
        server = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
        IPEndPoint ipep = new IPEndPoint(IPAddress.Any, port);
        server.Bind(ipep);
        server.Listen(4);
        AppendIncomingData(txtMsg, "Server Started..");

        btnStart.Enabled = false;
        lblStartDate.Text = DateTime.Now.ToString();
        server.BeginAccept(new AsyncCallback(AcceptConn), server);
    } 
    catch (Exception ex) 
    {
        AppendIncomingData(txtError, ex.Message);
    }
}

void AcceptConn(IAsyncResult ia)
{
    Socket client;
    try
    {
        Socket oldserver = (Socket) ia.AsyncState;
        client = oldserver.EndAccept(ia);

        StateObject state = new StateObject();
        state.workSocket = client;
        client.ReceiveTimeout = 1000;
        client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0, new AsyncCallback(ReceiveData), state);
        server.BeginAccept(new AsyncCallback(AcceptConn), server);
    }
    catch (Exception ex) 
    {
        AppendIncomingData(txtError, "AcceptConn Exception : " + ex.Message);
    }
}

/// <summary>
/// StateObject Class to read data from Client
/// </summary>
public class StateObject
{
    // Client  socket.
    public Socket workSocket = null;
    // Size of receive buffer.
    public const int BufferSize = 512;
    // Receive buffer.
    public byte[] buffer = new byte[BufferSize];
    // Received data string.
    public StringBuilder sb = new StringBuilder();
    public Int64 DeviceId = 0;
}

void ReceiveData(IAsyncResult ia)
{
    StateObject state = (StateObject)ia.AsyncState;
    Socket client = state.workSocket;

    string sql = "";
    string stringData = "";
    try
    {
        recv = client.EndReceive(ia);
        if (recv == 0)
        {
            client.Close();
            server.BeginAccept(new AsyncCallback(AcceptConn), server);
            return;
        }

        Array.Resize(ref state.buffer, StateObject.BufferSize);
        client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0, new AsyncCallback(ReceiveData), state);
    } //End Try
    catch (Exception ex)
    {
        AppendIncomingData(txtError, "ReceiveData Error : " + stringData + ": " + ex.Message);

        client.Close();
        server.BeginAccept(new AsyncCallback(AcceptConn), server);
    }
}

public void AppendIncomingData(object dataControl, string str)
{
    if (btnMsg.Text.ToUpper().Contains("START") == true)
        return;

    TextBox lst = (TextBox)dataControl;

    if (lst.InvokeRequired)
    {
        AppendIncomingData_Delegate method = new AppendIncomingData_Delegate(AppendIncomingData);
        lst.Invoke(method, new object[] { lst, str });
    }
    else
    {
        msglines += 1;
        if (msglines > 500)
        {
            msglines = 0;
            //lst.Clear();
            txtError.Clear();
            txtMsg.Clear();
        }
        lst.AppendText(Environment.NewLine + DateTime.Now.ToString() + " " + str);
    }
}

What could be the problem?

Answer 1

The symptom you describe is one of monotonically-increasing memory usage. That is, your program allocates more and more memory over time. Eventually, so much memory is allocated that some of it has to be swapped to disk. But if all of the allocated memory is still being used regularly, that memory has to be swapped back into RAM and other memory has to be swapped out. It's known as "thrashing".

Without a good, minimal, complete code example that reliably demonstrates the problem, it's impossible to know for sure what the problem is. But there are a couple of obvious issues in your code that could be the cause:

1. You call BeginAccept() when closing connections, in addition to having called it any time you open a connection. This means that over time, you create more and more outstanding accept I/O operations.

The above issue may be relatively minor, though without knowing for sure the pattern of client behavior it's impossible to know for sure. Because of the geometric nature of this problem, over time it could easily add a large amount of memory allocations to the process.

There is also this…

2. You appear to maintain a TextBox control, into which you append status messages related to the operation of your server (e.g. exceptions and client-connection notifications). You've commented out the line that would clear the TextBox in question. While you do clear the txtError and txtMsg objects, notably your receive completion callback method doesn't do anything with the received data, which suggests you've left out that detail in your code example for some reason. So it's possible you are also writing to some other TextBox as you received data, and are not clearing that one.

There is also the possibility that you are not discarding the state objects for closed sockets. Again, the code you posted doesn't do anything with received data, which is pretty nonsensical for any server, so presumably you left that detail out of the code example, leaving open the possibility that you are mismanaging that data structure (e.g. keeping all of the client data structures even when the client has disconnected).

For an answer less vague than the above, you need to post a better, less vague code example.

Finally, I will point out some other apparent bugs in the code:

1. You simply close the socket when the remote endpoint starts a shutdown of the connection. You should do a graceful closure on your end too (i.e. call Socket.Shutdown()) so that the remote endpoint sees a proper shutdown instead of a reset connection.
2. As I mentioned, you are calling BeginAccept() on socket errors and connection closures. Don't do that.
3. You are calling Array.Resize() after receiving data. There would be no reason to do this even in normal network receive situations, i.e. where you're actually resizing the array. But to call Array.Resize() by passing the same value for the new size as the current size of the array? Pointless.

Answer 2

It seems that AcceptConn keeps calling itself in the following line: server.BeginAccept(new AsyncCallback(AcceptConn), server); thereby creating an infinite loop. This would be so because none of the calls to EndAccept, BeginReceive or BeginAccept block.

void AcceptConn(IAsyncResult ia)
{
    Socket client;
    try
    {
        Socket oldserver = (Socket) ia.AsyncState;
        client = oldserver.EndAccept(ia);

        StateObject state = new StateObject();
        state.workSocket = client;
        client.ReceiveTimeout = 1000;
        client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0, new AsyncCallback(ReceiveData), state);
        server.BeginAccept(new AsyncCallback(AcceptConn), server);
    }
    catch (Exception ex) 
    {
        AppendIncomingData(txtError, "AcceptConn Exception : " + ex.Message);
    }
}