0

When I use php to

exec('sort /var/www/website/file_in.txt -o /var/www/website/file_out.txt');

the file that is output from the sort command is trunacted

input filesize= 2,442,541

output filesize= 1,146,881

I also noticed, when using php function filesize(file_in.txt), the returned value = 1,146,881, not the correct size as shown in a terminal session. I did call clearstatcache() prior to calling filesize().

Interestingly, filesize reports the size of the file_in as the same value that file_out is truncated to.

I am running a Linux _x86_64 64bit PHP version, so I thought that eliminates the 32bit filesize limitation issue for files gt 2MB.

When I run the sort command in a terminal session as user www-data, the output file is the same size as the input file, no truncation.

I tried writing a shell script to call from exec, hoping it would bypass a possible php buffer limit, but it has the same truncated output file.

I have TOP running in a separate terminal to watch CPU and MEMORY usage, but since the coammdn when run in a terminal does not truncate the output, this appears to be a PHP issue.

Is there any kind of obscure configuration .ini setting that I should check to solve this issue?

Thanks

Additional Info: I realized the output file is not being truncated, but rather, the input file is not being fully read in until EOF.

Steve Wasiura
  • 768
  • 1
  • 8
  • 19
  • 1
    May be this help: http://www.computerhope.com/unix/usort.htm, find sentence "SIZE may be followed by the following multiplicative suffixes" – Taron Saribekyan Jan 28 '17 at 19:37
  • thanks for your suggestion, but that webpage doesn't even list the option -S (buffer-size), which is what SIZE relates to. However, when investigating your suggestion, I found this webpage: https://www.gnu.org/software/coreutils/manual/html_node/sort-invocation.html , which gives a better description of buffer-size, so your effort was helpful. I will try the buffer size test next. I also learned there is a --debug option, which may help understand the truncation. – Steve Wasiura Jan 28 '17 at 20:18
  • N.B> --debug cannot be used at the same time -o output-file is being used. debug uses stndout(i.e. a terminal session screen) – Steve Wasiura Jan 28 '17 at 20:41
  • May be you can debug using terminal screen to get the problem, because I think problem connected with memory and it will be caused in terminal mode also. – Taron Saribekyan Jan 28 '17 at 20:55
  • i did sort in terminal mode and it worked without error. i did debug in terminal and no major issues were reported. but I have the solution now – Steve Wasiura Jan 28 '17 at 21:03
  • Please write an amswer on your qustion if you will solve the problem – Taron Saribekyan Jan 28 '17 at 21:16

1 Answers1

1

this appears to have been caused by a lack of available memory at this point of the php script's execution.

through trial tests, I increased the SORT command's option --buffer-size=4K starting at 4K, 5K, 10K, 40K, but nothing did the trick.

I was watching TOP to see how the CPU % memory usage was reported.

I did not think this was a problem so I did not describe it in my original question, but prior to this step of calling SORT thru EXEC, I was calling pdftotext thru EXEC. As that process was running, the server's CPU usage spiked up to 98%. Memory probably also spiked, but TOP's refresh rate did not capture it.

I imagined that I could add a sleep(5) before the SORT command was called, to pause the PHP script's execution, giving the CPU and Memory spike some time to return to normal. That solved the problem of SORT reading the entire input file and outputting all it's contents. This also solved the incorrect filesize() result.

In a production environment, I will spin-up a server with more capacity, and try to eliminate the sleep(5) delay. I can't wait until I get to startup level "ramen noodles" :)

Steve Wasiura
  • 768
  • 1
  • 8
  • 19