unable to do progressive rendering

Question

Years ago I wrote CGI programs at previous jobs that relied on progressive rendering, since those CGI programs could take a long time (minutes) to run, producing a line of output approx every second. I find that today, I cannot get progressive rendering to occur even with the simplest example.

I've seen a lot of suggestions on this topic about where to put CSS, scripts, etc. However, the trivial example below has none of that.

I don't see anywhere where browsers have an option to affect progressive rendering. I have tried this on several systems/devices with several browsers (chrome, firefox, opera), all with the same result.

Below is a trivial example that I expect to produce some output every 2 seconds, but instead it renders when the whole document is complete. Am I missing something obvious?

#!/usr/bin/env perl

select(STDOUT); $| = 1;     # don't buffer stdout

print "Content-Type: text/html\; charset=ISO-8859-1\n\n" ;
print "<html> <head> <title> Testing </title> </head> <body>\n" ;

my $message = "<code>" .
    "Why doesn't this render immediately? <br>\n" x 5 .
    "</code>\n" ;

for ( my $i=0 ; $i < 5 ; $i++ ) {
    print "$message\n" ;
    sleep(2) ;
}
print "</body></html>\n" ;

Try adding `Transfer-Encoding: chunked` https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding — Andy Ray, Dec 31 '16 at 03:33
Thanks. However, it didn't work for me. I tried a variant of what I found in http://stackoverflow.com/questions/17153009/chunked-transfer-encoding-browser-experience with a sleep between the 2 chunks, and STDOUT buffering disabled, and it still waited till the entire document was received. — RJ White, Dec 31 '16 at 09:30
Something is very wrong. The docs say the length is hex, and it will only work with hex, until you get to a 26 byte string when the hex length goes from 19, which works for a 25 byte string, to 1a (or 1A). Then it craps out with 'ERR_INVALID_CHUNKED_ENCODING'. sigh. — RJ White, Dec 31 '16 at 10:09

score 2 · Accepted Answer · answered Dec 31 '16 at 20:27

Your web server is likely buffering the response. $| = 1; sets up STDOUT to be autoflushed each time you print, eliminating the effects of the buffering in your script, but you also need to consider buffering that's happening in your web server.

There's no command or character sequence to flush the buffer, but you could simply just send a sufficient amount of data to fill the buffer so that it flushes on its own.

Just send inconsequential content, like a bunch of spaces:

print " " x 1024 * 8;

How much data you need to send depends on how large the buffers are configured to be in your web server. Typical buffer sizes are 4KiB or 8KiB, but be aware that if your server gzips your script's response then you will need to print considerably more (perhaps around 8MiB of space characters) in order to fill the server's buffers because the buffers will be filled with the compressed response.

Of course, you could also just disable the buffering in your server. How you do that depends on the web server. For nginx, take a look at X-Accel-Buffering.

That wasn't it, but it set me on the right track to find the issue, which I detailed in a followup answer. So I'll credit you for the solution. Thanks!! — RJ White, Jan 01 '17 at 12:12

score 1 · Answer 2 · edited May 23 '17 at 11:45

The response from ccm didn't work for me, but it set me on the right track to find the problem. The solution was to add the following to my Apache config:

SetEnvIfNoCase Request_URI \.cgi$ no-gzip dont-vary

which I found from Prevent output buffering with PHP and Apache

While trying out what @ccm suggested, it appears the buffer size is 1K, which is fine with me.

Thanks a huge amount to @ccm for setting me on the right path!

score 0 · Answer 3 · answered Dec 31 '16 at 10:23

You also need your webserver to keep the CGI script running long enough. A default Apache has a 1 minute timeout.

You already turned off buffering with $_, which is fine. There is no more control you can have from the script. The connection needs to stay open, even if you want to do chuncked transfer, which isn't really needed in your example.

It's the server who shuts it down after a while. And once the connection is gone the webserver doesn't send the remainder of your response over the wire because that connection is gone and the CGI handle is detached, so nothing is reading your output to pass it along.

Conclusion: Set the timeout to a higher value.

Anecdote: I used to work on a system with a timeout setting of about an hour, where a CGI based back office application did large DB queries over a huge MySQL database and stuff took long. People who worked with that tool typically started it and went to grab coffee or have lunch.

Thanks, but I already know about Apache timeouts and they aren't relevant in this case. This example should be a few seconds. The examples I am using for illustration are a maximum of 10 seconds with the sleeps I am using in order to witness whether immediate rendering is working or not. The connection is staying open - it waits until the entire document is complete before displaying anything. I have written CGI scripts in the past, with Apache timeouts set to 5 minutes or more, and everything was fine and I had progressive rendering. Something has changed and I don't know what. — RJ White, Dec 31 '16 at 11:27

unable to do progressive rendering

3 Answers3