0

I have a FastCGI script which starts creating a file for request and sends redirection header to the client with location of a new file just after a first pice of that file has been written to the HDD. Everything works fine but when client request a file from a redirection, it receives Content-lenght header with length of the file which has already been written to the file when request has been made and the client doesn't receive part of the file which was created after the request.

How can I configure Apache to avoid sending Content-lenght header for uncompleted files and keep sending files until it sends whole completed file?

Regards,

Michal Pietras.

Michal Pietras
  • 101

3 Answers3

0

I'm not sure if I'm following your question, but it sounds like you have a process that starts writing to File A sends a 3xx-series redirect message to the client telling it to go get File A, then continues writing to File A.

If that's what's going on your client is probably requesting File A before it it's done being written to, and it's getting the correct content-length for the contents of the file when it was requested.
If you want to send the whole file you need to finish writing to it before you issue the redirect.

voretaq7
  • 79,879
  • 17
  • 130
  • 214
  • you are following my question. File creation can take a few minutes. Client should receive file slowly just after new peace is created rather than waiting for file is completed and get it fast. I think I can say it is a little bit like a streaming. – Michal Pietras May 23 '11 at 18:06
  • You may want to forego the redirect then - Simply stream the data to the client from the main script. Other alternatives include redirecting the client to a CGI script that can stream the file rather than to the file itself, or sending the client a URL and a time to pick up the file when it's done, or some AJAX magic if this is happening in a browser, etc... – voretaq7 May 23 '11 at 18:54
  • Printing a file from the script is a good idea. What about performance of such operation in comparison to sending a file with traditional URL? – Michal Pietras May 23 '11 at 19:03
  • See my note on @DerfK's answer - I'd be more worried about HTTP Timeout than performance, particularly since printing directly will work (where "work" = "send the whole file, as long as you don't hit any time limits"), while the current setup doesn't :-) – voretaq7 May 23 '11 at 19:08
0

I haven't tested it, but try using mod_header:

Header unset Content-Length

You can put this inside a Directory section if the incomplete files are all in the same directory, or some other block if more appropriate.

Do note this breaks RFC behaviour and I'm not sure whether Apache would indeed be able to transfer the whole file after removing the header - for all we know it may still use an internal counter independent of this particular header.

Eduardo Ivanec
  • 14,881
  • 1
  • 37
  • 43
  • I will try it but I am not sure how Apache can determine when file is completed and nothing will appended to it. – Michal Pietras May 23 '11 at 18:47
  • @Michael It really can't determine that - It's likely to scan until it hits an EOF and then send what it saw. If the EOF moves later (because more data is appended) your client won't get the new data because Apache is already done serving the request... – voretaq7 May 23 '11 at 18:58
  • @voretaq7, I expected it. – Michal Pietras May 23 '11 at 19:22
0

According to the answer here, you have to change apache's source code to do it. According to the question and answer here, if you enable gzip compression in apache, it will have the side effect of disabling the content-length header since apache won't know the length until it is done compressing, and by then it has already started sending the compressed data.

Either way, I do not think what you are trying to do will work, unless your "first piece of that file" is so large that Apache cannot possibly read to the end of it (Apache's buffer, your kernel's TCP Send buffer, your network's buffer, and so on) before you write another large chunk of data to it. Also note that if apache is using sendfile() (and it almost certainly is on Linux unless you use gzip or SSL or you turn it off) then the length of the data to send is provided to the sendfile() function, and that will be the length of the file at the time apache starts sending it.

Community
  • 1
DerfK
  • 19,493
  • 2
  • 38
  • 54
  • @DerfK, Alternatively, I can "print" created file from FastCGI script instead of sending redirection. How does it impact performance? How much faster is passing a URL to file than "printing" it using simple FastCGI script? – Michal Pietras May 23 '11 at 18:51
  • The only problem I can foresee with printing the data directly from your CGI script is that if it takes a while to build the file you may exceed the HTTP Request Timeout limit in Apache (normally 5 minutes I believe). – voretaq7 May 23 '11 at 19:00
  • Can I somehow change this timeout? What about big files send over slow network - it can takes a few hours? – Michal Pietras May 23 '11 at 19:22
  • @Michal Depending on the language, there may be several kinds. If you send no data at all, your request could time out at Apache, and possibly even at your or your user's firewall. As long as data is moving, the firewalls should not time out. PHP (and possibly other script languages/environments) has a "maximum execution time" timeout that will kill the script whether it is sending data or not. You'll need to tell us more about your server and your programming language for us to tell you what you need to look for. – DerfK May 23 '11 at 21:29
  • I found in mod_fcgid that default timeout is 300s but I can amend it. I use C and I don't have any restriction set on firewall - I don't use it at all. Server is on FreeBSD. I guess if Apache or rather mod_fcgid won't kill my application everything should be fine. What exactly do you need to know? – Michal Pietras May 23 '11 at 22:04
  • @Michal That should cover it. If you're writing your CGI in C then unless you have some special CGI library that might have some timeout setting, your program won't have a timeout of its own. For `mod_fcgid` you will need to make `FcgidBusyTimeout` higher than it could take for the file to be sent (some timeouts say they can be set to 0 to disable, but this one does not). You will need to test and see what happens if a user cancels the download (I assume apache tells fcgid to tell your program this somehow, PHP has an option to ignore "user aborts"). – DerfK May 23 '11 at 23:14
  • I am concern about what happened when user cancels request too. I can't find anything stated explicitly in documentation but most likely should occur some stream error. Thanks for help. – Michal Pietras May 24 '11 at 09:10