splice() from pipe to TCP buffered?

Question

xpost from linuxquestions.org, sorry...

I wrote a small test program to see if a simple proxy would benefit from using splice() but it always takes 200ms for the data that I spliced from a pipe to a TCP socket to be read from the other end of the socket.

Here is the Perl program to test it:

package test_pipes_2;
use strict;
use warnings;
use IO::Handle();
use POSIX qw(:errno_h);
use Time::HiRes qw(time);
use English qw(-no_match_vars);
use IO::Socket::INET;
use Socket qw(IPPROTO_TCP TCP_NODELAY);

__PACKAGE__->run if not caller;

sub run {
    my( $class ) = @ARG;

    pipe my $parent_reader, my $child_writer;
    my $parent_reader_fd = fileno $parent_reader;
    my $child_writer_fd  = fileno $child_writer;
    pipe my $child_reader,  my $parent_writer;
    my $child_reader_fd  = fileno $child_reader;
    my $parent_writer_fd = fileno $parent_writer;

    my $server = IO::Socket::INET->new(
        LocalAddr   => '127.0.0.1:9000',
        Type        => SOCK_STREAM,
        Listen      => 5,
        ReuseAddr   => 1,
        Blocking    => 1,
    ) || die $OS_ERROR;

    my $client_browser = IO::Socket::INET->new(
        PeerAddr   => '127.0.0.1:9000',
        Type        => SOCK_STREAM,
        Blocking    => 1,
    ) || die $OS_ERROR;
#   setsockopt $client_browser, IPPROTO_TCP, TCP_NODELAY, 1;
    my $client_browser_fd = fileno $client_browser;

    my $server_browser = $server->accept() || die $OS_ERROR;
#   setsockopt $server_browser, IPPROTO_TCP, TCP_NODELAY, 1;
    my $server_browser_fd = fileno $server_browser;

    for( 1 .. 3 ) { #  100_000
        syswrite $client_browser, "HTTP/1.1 GET /foo/\r\nLocation: foo.com\r\n\r\n";
        $server_browser->recv( my $request1, 4096, POSIX::MSG_PEEK );

        syscall 313, $server_browser_fd, undef, $child_writer_fd, undef, 4096, 1;
        sysread $parent_reader, my $request2, 4096;

        my $response = "200 OK\n<head><title>html</title></head><body>body from $PID</body>";
        syswrite $parent_writer, $response;
        syscall 313, $child_reader_fd, undef, $server_browser_fd, undef, 4096, 1;
#       syswrite $server_browser, "\n"; # eliminates delay, adds syscall...
        sysread $client_browser, my $response1, 4096;
#       chomp $response1;

        if( $response1 ne $response ) {
            warn 'Got wrong response: ', $response1 // 'undef';
            no warnings 'once';
            $DB::single = 1;
        }
    }
}

1;

And here is a sample of the strace output:

socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 8 <0.000010>
setsockopt(8, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0 <0.000006>
bind(8, {sa_family=AF_INET, sin_port=htons(9000), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 <0.000008>
listen(8, 5)                            = 0 <0.000008>
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 9 <0.000006>
connect(9, {sa_family=AF_INET, sin_port=htons(9000), sin_addr=inet_addr("127.0.0.1")}, 16) = 0 <0.000037>
accept(8, {sa_family=AF_INET, sin_port=htons(49361), sin_addr=inet_addr("127.0.0.1")}, [16]) = 10 <0.000007>
write(9, "HTTP/1.1 GET /foo/\r\nLocation: foo.com\r\n\r\n"..., 41) = 41 <0.000014>
recvfrom(10, "HTTP/1.1 GET /foo/\r\nLocation: foo.com\r\n\r\n"..., 4096, MSG_PEEK, {sa_family=0x8b60 /* AF_??? */, sa_data="(\10\2\0\300\321\177\0\0\1\0\0\0\0"...}, [0]) = 41 <0.000007>
splice(0xa, 0, 0x5, 0, 0x1000, 0x1)     = 41 <0.000007>
read(4, "HTTP/1.1 GET /foo/\r\nLocation: foo.com\r\n\r\n"..., 4096) = 41 <0.000005>
write(7, "200 OK\n<head><title>html</title></head><body>body from 14019</body>"..., 67) = 67 <0.000005>
splice(0x6, 0, 0xa, 0, 0x1000, 0x1)     = 67 <0.000007>
read(9, "200 OK\n<head><title>html</title></head><body>body from 14019</body>"..., 4096) = 67 <0.200331>
write(9, "HTTP/1.1 GET /foo/\r\nLocation: foo.com\r\n\r\n"..., 41) = 41 <0.000016>
recvfrom(10, "HTTP/1.1 GET /foo/\r\nLocation: foo.com\r\n\r\n"..., 4096, MSG_PEEK, {sa_family=0x8b60 /* AF_??? */, sa_data="(\10\2\0\300\321\177\0\0\1\0\0\0\0"...}, [0]) = 41 <0.000008>
splice(0xa, 0, 0x5, 0, 0x1000, 0x1)     = 41 <0.000006>
read(4, "HTTP/1.1 GET /foo/\r\nLocation: foo.com\r\n\r\n"..., 4096) = 41 <0.000005>
write(7, "200 OK\n<head><title>html</title></head><body>body from 14019</body>"..., 67) = 67 <0.000005>
splice(0x6, 0, 0xa, 0, 0x1000, 0x1)     = 67 <0.000006>
read(9, "200 OK\n<head><title>html</title></head><body>body from 14019</body>"..., 4096) = 67 <0.200622>
write(9, "HTTP/1.1 GET /foo/\r\nLocation: foo.com\r\n\r\n"..., 41) = 41 <0.000018>
recvfrom(10, "HTTP/1.1 GET /foo/\r\nLocation: foo.com\r\n\r\n"..., 4096, MSG_PEEK, {sa_family=0x8b60 /* AF_??? */, sa_data="(\10\2\0\300\321\177\0\0\1\0\0\0\0"...}, [0]) = 41 <0.000007>
splice(0xa, 0, 0x5, 0, 0x1000, 0x1)     = 41 <0.000006>
read(4, "HTTP/1.1 GET /foo/\r\nLocation: foo.com\r\n\r\n"..., 4096) = 41 <0.000005>
write(7, "200 OK\n<head><title>html</title></head><body>body from 14019</body>"..., 67) = 67 <0.000005>
splice(0x6, 0, 0xa, 0, 0x1000, 0x1)     = 67 <0.000005>
read(9, "200 OK\n<head><title>html</title></head><body>body from 14019</body>"..., 4096) = 67 <0.200649

Note the ~200ms read(9,...) calls. If I uncomment the line to send a "\n" then there is no delay. What am I doing wrong? Thanks!

What happens if you uncomment the lines setting `TCP_NODELAY`? — caf, Sep 22 '10 at 03:03
It runs identically even when TCP_NODELAY is set. It's behaving as if I had set TCP_CORK on the $server_browser socket. — bennymack, Sep 22 '10 at 12:45
After more code reading and experimentation I determined that if I change the len of the second call to splice to be the length of the response it does not have a 200ms pause. However in my real world usage of this, the send and splice would happen in two separate processes making it difficult to splice exactly the amount that was sent. It seems that this is not a problem with the common usage of sending a file since it's trivial to know the file size... — bennymack, Sep 24 '10 at 19:33
Of course I find this a week later... http://www.spinics.net/lists/netdev/msg84138.html — bennymack, Sep 30 '10 at 19:14

splice() from pipe to TCP buffered?

0 Answers0