5

I have a perl script, that goes through a couple of gig worth of files and generates a report.

In order to calculate percentile i am doing the following

my @values = 0;
while (my $line = <INPUTFILE>){
    .....
    push(@values, $line);

}
# Sort
@values = sort {$a <=> $b} @values; 

# Print 95% percentile
print $values[sprintf("%.0f",(0.95*($#values)))];

This obviously saves all the values upfront in a array and then calculates the percentile, which can be heavy on memory (assuming millions of values), is there a more memory efficient way of doing this?

ZOXIS
  • 578
  • 5
  • 15

1 Answers1

3

You can process the file twice: the first run only count the number of lines ($.). From that number, you can count the size of the sliding window which will only keep the highest numbers needed to find the percentile (for percentiles < 50, you should invert the logic).

#!/usr/bin/perl
use warnings;
use strict;

my $percentile = 95;

my $file = shift;
open my $IN, '<', $file or die $!;

1 while <$IN>;             # Just count the number of lines.
my $line_count = $.;
seek $IN, 0, 0;            # Rewind.

# Calculate the size of the sliding window.
my $remember_count = 1 + (100 - $percentile) * $line_count / 100;

# Initialize the window with the first lines.
my @window = sort { $a <=> $b }
             map scalar <$IN>,
             1 .. $remember_count;
chomp @window;

while (<$IN>) {
    chomp;
    next if $_ < $window[0];
    shift @window;
    my $i = 0;
    $i++ while $i <= $#window and $window[$i] <= $_;
    splice @window, $i, 0, $_;
}
print "$window[0]\n";
choroba
  • 231,213
  • 25
  • 204
  • 289
  • The inner `while` may be more understandable as `$i++ while $window[$i] < $_ and $i < $#window` (is that equivalent?). – amon Aug 13 '13 at 12:43
  • @amon: Probably, maybe with a 1 off error. TITS - Try It To See :-) – choroba Aug 13 '13 at 12:47
  • I took the liberty of editing in a `chomp @window` and the simplified loop. Changing the order of tests led to a drastical decrease in warnings. – amon Aug 13 '13 at 13:03
  • @amon: I fixed the typo you made. I was getting no warnings. – choroba Aug 13 '13 at 13:14
  • Thanks Choroba, that indeed took care of the high memory consumption. however it is now slower then before. i guess you cant have everything. – ZOXIS Aug 13 '13 at 13:27