Generating Entropy on Multiple Arrays and output to MySQL

Question

I have a program that produces a hash of arrays based on MySQL data. Each array has numerical values in it. Using Perl, how do I generate the entropy of each array and output the results in a seperate MySQL table? The new table should have the columns:

 ID         Array           Entropy
 -----      -----           -----
 1          topic(key)      entropy of all values belonging to the topic

Here is the current program that generates the hash of arrays:

my %values_by_topic; 
my $sth = $dbh->prepare('SELECT Topic, Value FROM Table'); 
$sth->execute(); 
while (my $row = $sth->fetch()) {    
   my ($topic, $value) = @$row;     
   push @{ $values_by_topic{$topic} }, $value; 
}

Perhaps you should elaborate a bit more on what kind of output you need. As for myself, I've no idea what you mean by "entropy". — TLP, Sep 20 '12 at 18:51
Without getting too complicated ... entropy will tell you how predictable a set of data is. If your dataset consisted of the same number over and over, your entropy would be 0. The more random your data, the higher the entropy. — user1652974, Sep 20 '12 at 19:46

score 1 · Answer 1 · answered Sep 20 '12 at 20:06

You asked two question:

How do I generate the entropy of each array?
How do I output the results in a separate MySQL table?

The first is for you to tell us and for us to help you implement.

The second is answered below.

my $sth_select = $dbh->prepare('
   SELECT Topic,
          Value
     FROM Table1
    ORDER BY Topic
');

my $sth_insert = $dbh->prepare('
   INSERT INTO Table2
             Topic,
             Entropy
          ) VALUES (
             ?, ?
          )
');

my $last_topic;
my @values;
$sth_select->execute();
while (my $row = $sth_select->fetch()) {
    my ($topic, $value) = @$row;

    if (@values && $topic ne $last_topic) {
        my $entropy = calculate_entropy(@values);
        $sth_insert->execute($last_topic, $entropy);
        @values = ();
    }

    $last_topic = $topic;
    push @values, $value;
}

if (@values) {
    my $entropy = calculate_entropy(@values);
    $sth_insert->execute($last_topic, $entropy);
    @values = ();
}

Notes:

Some databases can't have two open statements handles for a given database handle. If MySQL is like that, just create two database handles.

Locking and transaction is left as an exercise to the reader. (Or someone can edit it in.)

Generating Entropy on Multiple Arrays and output to MySQL

1 Answers1

Notes: