Windows namedpipe maximum limit

Question

There's bug occurs in my project, and I found that it could be the large message passing by the windows namedpipe.

The process1 produce about 80 KB message, and I have a namepdpipe attach to its standard output.

After the process 2 read the message from the namedpipe, I found the message is incomplete.

The pseudo code is like that,

char buffer[4096] ;
string msg ;
while( !namedpipe.isEmpty() ) {
    int length = namedPipe.read(buffer, 4096) ;
    msg.append(buffer, length) ;
}

After google some information about the namedpipe, I found the namedpipe is limit in 65535 byte.

The 80KB message would exceed the limit.

But when I insert the Sleep(1000) before the read.

char buffer[4096] ;
string msg ;
while( !namedpipe.isEmpty() ) {
    Sleep(1000) ;
    int length = namedPipe.read(buffer, 4096) ;
    msg.append(buffer, length) ;
}

The message is OK and complete.

I think that in the sleeping moment, system requests the memory for the namedpipe.

So the namedpipe would only ensure 65535 byte to use.

Is the progress that I infered correct?

user207421 · Answer 1 · 2016-12-11T05:56:55.867

0

isEmpty() is not an end of message indicator.

You should change the loop termination condition to terminate when the expected message length is reached.

The maximum amount of data in the pipe is defined at creation time via the nInBufferSize and nOutBufferSize parameters to CreateNamedPipe. See the Remarks section there.

The sleeps are literally a waste of time.

edited Dec 11 '16 at 05:56

answered Dec 11 '16 at 05:46

user207421

305,947
44
307
483

I'm not sure if OP expects the pipe buffer to be larger or if he is actually talking about message size. 80K can be sent over a pipe in one WriteFile call regardless of what was passed for the buffer sizes in CreateNamedPipe. There used to be a limit for one write around 64K in Windows XP, and that may be what OP found. I wish I could upvote twice for "the sleeps are literally a waste of time" by the way. – Millie Smith Dec 11 '16 at 06:08
@MillieSmith The documentation I cited is quite clear. I don't see how it is possible for a write to exceed the buffer size, unless it is within the fudge factor. – user207421 Dec 11 '16 at 06:29
The documentation is for how big the internal buffer is, not how much data can be sent over at once. Sample server with asserts: http://pastebin.com/ciJj8BDX, sample client with asserts: http://pastebin.com/UTjZ2X8f. The WriteFile call doesn't return until all the data is sent (though the FlushFileBuffers call is still needed). This works with overlapped IO too and also works writing from client to server. Start the server, then the client. I was too lazy to properly wait for the named pipe to be created in the client. – Millie Smith Dec 11 '16 at 07:09
@MillieSmith If there is a difference between 'how big the buffer is' and 'how much data can be sent over at once' it eludes me, as does the meaning of 'the `WriteFile()` call doesn't return until all the data is sent' when '(though the `FlushFileBuffers()` call is still needed)'. The latter claim conflates sending with receipt by the client. I am relying on the MSDN documentation. What is your source? – user207421 Dec 11 '16 at 08:41
I'm not really sure on the FlushFileBuffers thing. I just know it's necessary to call that before calling DisconnectNamedPipe. Calling DisconnectNamedPipe right after WriteFile returns can cause the client to not get all the data. I need to spend more time in this area to fully understand it. There is a difference between "how big the internal buffer is" and "how much data can be sent via one write call". My sources are documentation, this SO link http://stackoverflow.com/a/33555632/2850543 and the source code (no pun intended) that I posted above and ran successfully on my machine. – Millie Smith Dec 11 '16 at 09:04
From the CreateNamedPipe docs: "Whenever a pipe write operation occurs, the system first tries to charge the memory against the pipe write quota. If the remaining pipe write quota is enough to fulfill the request, the write operation completes immediately. If the remaining pipe write quota is too small to fulfill the request, the system will try to expand the buffers to accommodate the data using nonpaged pool reserved for the process. The write operation will block until the data is read from the pipe so that the additional buffer quota can be released." (1/2) – Millie Smith Dec 11 '16 at 10:31
cont: "Therefore, if your specified buffer size is too small, the system will grow the buffer as needed, but the downside is that the operation will block. If the operation is overlapped, a system thread is blocked; otherwise, the application thread is blocked." (2/2) So the buffers actually do grow internally sometimes. – Millie Smith Dec 11 '16 at 10:32
Actually if you increase the client buffer size to 81920 bytes in my example, it receives the entire payload in exactly one `ReadFile` call on my machine. So that's exactly one `ReadFile` call from the client and one `WriteFile` call on the server to transfer 81920 bytes when only 4096 was passed to the `nInBufferSize` and `nOutBufferSize` parameters for `CreateNamedPipe` (20 times the buffer size was transferred at once). The `CreateNamedPipe` docs also note that it can ignore the buffer sizes passed in: "The input and output buffer sizes are advisory." – Millie Smith Dec 12 '16 at 01:34

Windows namedpipe maximum limit

1 Answers1