3

Using Winsock, C++, I send and receive the data with send()/recv(), TCP connection. I want to be sure that the data has been delivered to the other party, and wonder if it is recommended to send back some acknowledgment message after (if) receiving data with recv.

Here are two possibilities, and please advice which way to go:

  1. If send returns the size of passed buffer, assume that the data has been delivered at least to recv function on the other side of wire. When I say "at least", I mean even if the recv fails there (e.g. due to insufficient buffer, etc.), I don't care, I just want to be sure I've done my server part of work properly - I've sent the data completely (i.e. the data reached the other machine).

  2. Use additional acknowledgment: after receiving the data with recv, send back some ID of received packet (part of header of each data sent) signaling the successful receive operation of that packet. If I don't receive such "acknowledgment message" after some interval, return failure code from the sender function.

The second answer looks more safe, but I don't want to complicate the transfer protocol if it is redundant. Also please note that I'm talking about the TCP connection (which is more safe by itself than UDP).

Is there any other mechanisms (maybe some other APIs? maybe WSARecv()/WSASend() work differently?) of ensuring that the data was delivered to the recv function on the other side?

If you recommend the second way, could you please give me some code snippet that allows me to use recv with timeout to receive the acknowledgment? recv is a blocking operation so it will hang forever if the previous send attempt failed (the other party was not notified). Is there any simple way of using recv with timeout (without creating separate thread every time which would probably be the overkill for each and every send operation).

Also the amount of data I pass to send function might be quite big (several megabytes), so how to choose the timeout for "acknowledgment message"? Maybe I should "split" large buffers and use several send calls? I think it will get quite complicated, please advice!

EDIT: OK, you people are suggesting that TCP/IP stack will handle it (i.e. no manual acknowledgment required), but this is what I found on MSDN page: "The successful completion of a send function does not indicate that the data was successfully delivered and received to the recipient. This function only indicates the data was successfully sent." So even if the TCP mechanism has the ability to ensure data delivery, I can't get that status (success or not) via send() function, or any other Winsock function I know. Do you know any way of getting the status from the TCP layer? Again - return value of send() function seems to be not enough!

========================================================

EDIT 2: OK, I think we agree that even though TCP protocol considers the error handling when something goes wrong, the send() function of Winsock is not capable of reporting the errors (simply because it returns before actual transmitting of data starts by the network driver). So here is a million dollar question: Does the send() function of Winsock at least ensure that no other packets will be delivered to the other party until the current packet will be? In other words, if the sending fails for some network failure (but not reported by send() call), and then the network failure will be fixed before next call of send() function with next chunk of data, will it be ensured that the previous packet (which failed but not reported by send()) will be delivered before the next packet? In other words, is there a chance that the one particular send() function will fail "silently", so that subsequent send() calls will succeed but the first packet will be lost? AGAIN - I'm not talking at the TCP level, I'm talking at the Winsock API level!

TX_
  • 83
  • 1
  • 5
  • 2
    You are asking for a pony. TCP guarantees delivery. It will, it takes the send() buffer and returns immediately, promising to deliver it. But there's no great workaround for a 8.5 earthquake followed by a tsunami. Or somebody tripping over the power cord. Does that matter? Are there not bigger things to worry about when that happens? If you say "no!" then don't use TCP. – Hans Passant Jun 11 '11 at 23:40
  • It may be wise to remember that packets only travel at the speed of light. 50 ms travel time is not unusual. The OS **must** send the second packet before the first one arrives, if it wants to send more than 20 packets per second (~30 kB/s). Obviously the OS can't wait 100 ms for confirmation that the first packet arrived before sending the second packet. Your LAN may be faster, say 100 us, but waiting 200 us after each packet would limit your LAN to 5000 packets/s = 7.5 MB/s. Not great either. – MSalters Nov 04 '15 at 13:51

4 Answers4

3

Why don't you trust your TCP/IP stack to guarantee delivery. After all, that is the whole point of using TCP instead of UDP.

tster
  • 17,883
  • 5
  • 53
  • 72
  • So you mean the the return value of send() function is enough to assume the sending success? – TX_ Jun 11 '11 at 22:57
3

The existing answers here are mostly correct: if you use TCP you really don't need to worry about reliable delivery of your packets to your peer.

But this is a dangerous view for some systems where data integrity must be taken to the next level: the common criteria auditing requirement FAU_STG.4.1 requires the ability to prevent auditable events if the audit log might suffer a loss of audit entries. (For example, the Linux auditd(8) audit logging daemon can be configured to place the computer in single-user-mode or halt the system completely when there is no more space left for audit logs.) Audit logs from remote systems should probably be maintained until it is known that they have been successfully written to centralized log servers.

Financial transactions would probably be best handled with a more reliable protocol than simple TCP as well -- crediting or debiting accounts would be best handled with a multi-staged protocol to ensure availability of funds, perform the transaction, then report the result of the transaction to the origination point.

TCP allows nearly a gigabyte of in-flight data between two peers (under extreme conditions); depending upon the requirements of your application, you might need to maintain that data at the sending side until you receive positive confirmation from your peer that the data has been properly handled.

Thankfully, most applications aren't this critical; losing a megabyte of data here or there down a socket that reports a closed connection at some point "in the future" really isn't horrible -- we just re-try our HTTP request, or re-attempt the SFTP connection.

Update

A socket will only accept enough data to fill its available window. The window size is negotiated between the two peers during the session handshake. So your calls to send() will begin blocking when the socket's window fills. (The OS might keep letting you add data to its internal buffers too, but at some point the writes will block.) If the peer breaks the connection with a RST or ICMP Unreachable message, a future call to send() will return an error value for Connection Reset or Broken Pipe.

Update 2

I'm not talking at the TCP level, I'm talking at the Winsock API level

This might be the source of confusion. send() has no choice but to adhere to the TCP behavior when used with TCP.

TCP guarantees in-order reliable delivery of a stream of bytes, to the extent that packets can be delivered. (See @Hans's comment about a pony and careless people kicking power cords.) The peer program will see bytes in the correct order they were sent. (Well, okay, TCP also has out-of-band urgent packet delivery, but I haven't actually seen any applications that use it. Using OOB packets, you can get some data out-of-line. Forget I mentioned it.)

If the remote program receives a byte sent on a TCP stream, it reliably received all preceding bytes as well. (Well, there are entire classes of replay attacks that splice together legitimate and fake packets for the remote peer, but those are increasingly difficult on systems with randomized initial sequence numbers. If this is within your threat model, you should be using TLS on top of TCP to provide cryptographically strong tamper evident information. But TLS can't provide better per-packet delivery notification.)

sarnold
  • 102,305
  • 22
  • 181
  • 238
  • I think I'm getting the picture. Could you please view my last edit (of main question) and confirm that I can trust the mentioned behavior of Winsock? I.e. if one packet fails no other packets will be delivered before it is re-sent! – TX_ Jun 11 '11 at 23:55
  • +1, a nice summary. One thing to be aware of though is that it could well be a little different if you are using async send calls rather than potentially blocking ones, see here: http://www.serverframework.com/asynchronousevents/2011/06/tcp-flow-control-and-asynchronous-writes.html for detaisl. – Len Holgate Jun 12 '11 at 18:41
  • @Len, good article, thanks for passing it along. I'm curious though, is there really no API that could inform a program which sockets are _prepared_ to accept more data? The POSIX `select()` API makes it easy to avoid overstuffing the send buffers, and I'd hope something similar would be available on Windows to avoid hacky workarounds. – sarnold Jun 12 '11 at 23:41
  • SIO_IDEAL_SEND_BACKLOG_QUERY lets you know what the ideal amount to have queued is - but that's the subject of a yet to be written article. For highest overlapped performance you probably want to turn off the TCP stack send buffers anyway, set them to zero... In that case you're working purely on the TCP window that the client advertises. Besides, for optimum performance you NEED to have extra buffers already submitted and waiting; the trick is not to have too many :) – Len Holgate Jun 13 '11 at 07:19
1

If you use UDP and you care about the data actually being received by the other side you NEED to use ACK, but if you don't need the speed of UDP you should use TCP, as it does the ACKing for you.

Antwan van Houdt
  • 6,989
  • 1
  • 29
  • 52
  • So you mean the the return value of send() function is enough to assume the sending success? – TX_ Jun 11 '11 at 22:57
  • yes, if you are using a TCP connection the TCP connection takes care of the data being received or not, thats why TCP is so awesome :) – Antwan van Houdt Jun 11 '11 at 23:16
  • Please view my edit (of main question). Here is what we read at MSDN: "The successful completion of a **send** function does not indicate that the data was successfully delivered and received to the recipient. This function only indicates the data was successfully sent." So how to get that status (success or failure) from TCP layer itself? Seems that the return value of **send()** function is not enough! – TX_ Jun 11 '11 at 23:22
  • Only if the TCP connection is to be disconnected before your data has been delivered it wouldn't be -- don't worry about this if you are using TCP, and if you have an active connection you can always use a keepalive to test whether the other side is still alive and kicking! – Antwan van Houdt Jun 11 '11 at 23:42
  • My only wish is to ensure that no packet fails "silently", that is, if one packet fails no other packets will be delivered before it is re-sent! Please view my last edit. Please confirm if I can trust that functionality of Winsock. Thanks! – TX_ Jun 11 '11 at 23:52
  • Yes you can.. its TCP, they take care of that. – Antwan van Houdt Jun 12 '11 at 20:51
0

I think you are over complicating this, trust your TCP/IP software stack and the reliable delivery it offers. TCP sockets operate on streams of data, not packets. Also one call to send does not guarantee one call to recv.

Sam Miller
  • 23,808
  • 4
  • 67
  • 87
  • You are contradicting yourself. You say "one call to **send** does not guarantee one call to **recv**" so that I can't be sure just based on the return value of **send()**, right? So how do I "trust my TCP/IP software stack" than? i.e. how do I know if the operation failed? (Please view my edit as well). – TX_ Jun 11 '11 at 23:16