1

I'm using WinHTTP to transfer large files to a PHP-based web server and I want to display the progress and an estimated speed. After reading the docs I have decided to use chunked transfer encoding. The files get transferred correctly but there is an issue with estimating the time that I cannot solve.

I'm using a loop to send chunks with WinHttpWriteData (header+trailer+footer) and I compute the time difference between start and finish with GetTickCount. I have a fixed bandwidth of 4mbit configured on my router in order to test the correctness of my estimation.

The typical time difference for chunks of 256KB is between 450 - 550ms, which is correct. The problem is that once in a while (few seconds/tens of seconds) WinHttpWriteData returns really really fast, like 4-10ms, which is obviously not possible. The next difference is much higher than the average 500ms.

Why does WinHttpWriteData confirms, either synchronously or asynchronously that it has written the data to the destination when, in reality, the data is still being transferred ? Any ideas ?

Oversimplified, my code looks like:

while (dataLeft)
{
t1 = GetTickCount();
WinHttpWriteData(hRequest, chunkHdr, chunkHdrLen , NULL); 
waitWriteConfirm();
WinHttpWriteData(hRequest, actualData, actualDataLen , NULL); 
waitWriteConfirm();
WinHttpWriteData(hRequest, chunkFtr, chunkFtrLen , NULL); 
waitWriteConfirm();
t2 = GetTickCount();
tdif= t2 - t1;
}
D.Laurentiu
  • 11
  • 2
  • _WinHttpWriteData returns really really fast_ - what is the return value in this case? – zett42 Aug 18 '17 at 10:24
  • For every call the result is true indicating there were no errors. Also, no errors are reported in the callback function. – D.Laurentiu Aug 18 '17 at 14:41
  • If you need progress notifications, set up a callback using [WinHttpSetStatusCallback](https://msdn.microsoft.com/en-us/library/windows/desktop/aa384115.aspx). – IInspectable Aug 25 '17 at 20:18

1 Answers1

0

This is simply the nature of how sockets work in general.

Whether you call a lower level function like send() or a higher level function like WinHttpWriteData(), the functions return success/failure based on whether they are able to pass data to the underlying socket kernel to not. The kernel queues up data for eventual transmission in the background. The kernel does not report back when the data is actually transmitted, or if the receiver acks the data. The kernel happily accepts new data as long as there is room in the queue, even if it will take awhile to actually transmit. Otherwise, it will block the sender until room becomes available in the queue.

If you need to monitor actual transmission speed, you have to monitor the low level network activity directly, such as with a packet sniffer or driver hook. Otherwise, you can only monitor how fast you are able to pass data to the kernel (which is usually good enough for most purposes).

Remy Lebeau
  • 555,201
  • 31
  • 458
  • 770
  • I don't think your explanation is accurate. What happens if I pull the plug in that exact moment and the transfer does not finish ? The application regards the transfer as successful but the reality is different. The actual completion is called (or should be) after the data has successfully exited the outgoing interface and was physically written to the "wire", the kernel has seen this and it signals back to the userspace app. Otherwise, there could be false-positives confirmations which may lead to ugly assumptions in an application logic... – D.Laurentiu Aug 18 '17 at 16:20
  • @D.Laurentiu: if you pull the cable out, the kernel keeps accepting new data until the buffer fills up or the OS invalidates the connection, which may take awhile. TCP was designed to survive temporary network outages (from nuclear attacks). Short hiccups and auto-reconnects are not uncommon, so it may take awhile for a socket to timeout internally. Unless the interface actively reports its physical state to the OS so it can invalidate the connection immediately, which is not guaranteed. So it is not unheard of for apps to think connections are still sending data even with an unplugged cable. – Remy Lebeau Aug 18 '17 at 17:04
  • @D.Laurentiu: completion events are usually triggered when the kernel accepts data, not when data goes out over the wire. – Remy Lebeau Aug 18 '17 at 17:05
  • I agree that my issue is related to a buffer taking in data and reporting to the upper levels that the write operation is done. I also agree that TCP is designed for transparent retransmission, but it also signals if the retransmissions were successful or not. This is why it's reliable as opposed to udp, for example. Given this it should report to upper layers failed transmissions, as this is what it was designed for. In this situation, if the application has already received an "write-ok" confirmation, to whom will it report the error ? – D.Laurentiu Aug 18 '17 at 17:12
  • In my point of view, this issue is strictly related to WinHTTP and how it implements outgoing chunked transfer. I have never had any issues with other libraries such as libcurl... – D.Laurentiu Aug 18 '17 at 17:12