I answered your question on reddit, as well, but figured I'd reply here in case someone else is searching for this kind of thing.
You've got two problems with the way your server handles the reads. First of all, your client object's constructor:
Public Sub New(ByVal client As TcpClient)
'New client connects
Me.client = client
client.GetStream().BeginRead(New Byte() {0}, 0, 0, AddressOf Read, Nothing)
End Sub
BeginRead expects you to be collecting your data in a way that can be passed to the callback method; "Read" in this instance. Typically, that's done by creating a new object to hold the "State" of this async operation, and passing it in as the last parameter to BeginRead. Your method call here is creating a new byte array, but isn't holding a reference to it to pass in as the last argument to the method. That means that the data that gets read by this call will just disappear after being read, since it never gets passed into a method to store it.
Secondly, your Read operation:
Public Sub Read(ByVal ar As IAsyncResult)
Dim reader As New StreamReader(client.GetStream())
clientPacket &= reader.ReadLine()
client.GetStream().BeginRead(New Byte() {0}, 0, 0, AddressOf Read, Nothing)
End Sub
Since you're not passing in the data that was read in the async call, you're creating a new StreamReader here to pull more data in from the client. This is not an asynchronous method call. Your reader.ReadLine() call will block until a newline is encountered, at which point the data will be appended to clientPacket. You then call BeginRead again, with the same issue as noted above, meaning you'll lose more data.
In addition, you're never clearing your AsyncResult objects by invoking EndRead() on the stream, which will eventually result in resource starvation when the CLR runs out of worker threads for your async operations.
Here's an example of the way I'd implement this kind of task. It's in C# since that's what I'm most comfortable with, so sorry about that. ;)
Client Code
Server Code
I hope this helps!