Issue regarding string comparisons and references in perl

Question

Here is where I begin. I am reading arrays from a database one at a time using a while loop. I want to pick up on elements from the database that are duplicates (on certain fields). I want to keep only the items that are unique on these fields. Then I want to print out the data I have kept in a certain way. I created the code I thought would do it, but it gives me everything including items which are duplicates on the field. I've been searching and searching and I can't figure it out, I'm thinking, as a perl noob, I am missing something simple. Code is as follows:

my @uniques = ();
my $output;

while (my @itemArray = $sth->fetchrow_array() ) {
    my $duplicateFlag = 0;  
    foreach (@uniques){
        if(  ($itemArray[3] eq "$_->[3]") and ($itemArray[4] eq "$_->[4]")
               and ($itemArray[5] eq "$_->[5]" ) and ($itemArray[6] eq "$_->[6]" )
               and ($itemArray[7] eq "$_->[7]" ) and ($itemArray[8] == "$_->[8]" ) ){
            $duplicateFlag = 1;
        }
    }
    if( $duplicateflag == 0){
        $refToAdd = \@itemArray;
        push(@uniques, $refToAdd);
        $output .= "$itemArray[3]" . "\t$itemArray[8]" . "\t$itemArray[5]" . "\t$itemArray[7]\n";
    }
}
print $output

the code looks reasonable; can you show data that makes it misbehave? ideally like output from `Data::Dumper::Dumper({uniques=>\@uniques,refToAdd=>$refToAdd})` from just after the $refToAdd assignment for a row that should have been treated as a duplicate but wasn't — ysth, May 09 '11 at 22:00
You may wish to use this advice to do the deduplication in SQL. It's simpler and far more efficient. http://stackoverflow.com/questions/306743/how-to-detect-duplicate-rows-in-a-sql-server-table — Schwern, May 09 '11 at 22:57
@Schwern that looks like an excellent way to do it. I was trying to compensate for my super noobness in SQL with my noobness in perl. I might try to implement it using that SQL statment later. I literally hadn't seen a single SQL statement until 2 days ago so this is all a little new. Thanks a lot for offering up an excellent strategy. — Brian, May 10 '11 at 02:06

score 5 · Answer 1 · answered May 09 '11 at 21:00

One possibility: Use hashes to determine whether or not an item has been seen before. A bit simplified from your code:

my %dupHash;
while (my @itemArray = $sth->fetchrow_array() ) {
    my $uniqueItem = itemArray[4];
    if (not exists $dupHash{$uniqueItem}) {
        print "Item $uniqueItem\n";
        $dupHash{$uniqueItem} = \@itemArray;
    }
}

Okay, it's very simplified, but you get the idea. By using a hash with the values I want to verify are unique, I can avoid the double loop and the O² algorithm efficiency. (Dang! All those years in college finally paid off!).

You'll probably want to use a more complex hash key by combining all the fields you want to search for dups on. Maybe something like this:

 # Probably could use join to make it more efficient...
 my $uniqueKay = "$item[3]:$item[4]:$item[5]:$item[6]:$item[7]:$item[8]";
 if (not exists $dupHash{$uniqueKey}) {

The main thing is avoiding looping through all unique items again and again if you can store them in a hash.

My issue had to do with capitalization, so that's fixed, whew!! Anyway, after I got it working, I thought I'd try to reimplement it this way, and got it working no problem. I really like this strategy, I had seen multiple place people say use a hash for duplicates, but I didn't get how I could use it for my specific problem. I would've never thought toput the items together like this, honestly i overlooked that. Thanks a lot for pointing this out. — Brian, May 10 '11 at 02:02
You should be adding [use strict](http://perldoc.perl.org/strict.html) and [use warnings](http://perldoc.perl.org/warnings.html) in your scripts. That'll help you catch these capitalization errors. Using hashes to track already set data is an easy and fast way to avoid duplications. — David W., May 10 '11 at 03:05

score 4 · Answer 2 · answered May 09 '11 at 20:23

4

Possibly:

$itemArray[8] == "$_->[8]"

should be:

$itemArray[8] eq "$_->[8]"

to match all the others.

Another thing that may solve your problem is removing the quote marks around "$_->[8]". Depends what your data are.

answered May 09 '11 at 20:23

ADW

4,030
17
13

I think == or eq will work as this field happens to be numerical. I just chose == because of that. – Brian May 10 '11 at 00:28
The smart match operator can perhaps be used somehow.. with that you can do `if (@a1 ~~ @a2)` – TLP May 10 '11 at 09:24

TLP · Accepted Answer · 2011-05-09T23:15:18.857

You are getting all the duplicates because $duplicateflag is undefined at line 13. Running a syntax test on your script with use strict; use warnings; on produces the following warning:

Global symbol "$duplicateflag" requires explicit package name at t10.pl line 18.

And if we scrutinize your definition of "that" variable, it says:

my $duplicateFlag = 0;

Which is to say, you have a capital F, which means $duplicateflag is not the same variable as $duplicateFlag. The check undef == 0 still produces a true value and causes a false positive.

To avoid problems like this, always run your scripts with

use strict;
use warnings;

Wow, I can't believe I missed that. Thanks so much! I've learned my lesson about not using strict and warning. — Brian, May 10 '11 at 01:40

score 1 · Answer 4 · answered May 10 '11 at 12:08

SQL group by or select distinct is the SQL database way of keeping rows unique.

But if you're going to do this in Perl, I agree that hashes and keys are the way to go. However, any delimiter we could suggest, might also be there in the data. That gives you the potential for an ambiguous match. One hash-based method is unequivocal and uses Perl's natural structures to delimit your fields.

That is why I present the following.

my %uniq;

while ( my @r = $sth->fetchrow_array()) {
    next unless $uniq{ $r[3] }{ $r[4] }{ $r[5] }{ $r[6] }{ $r[7] }{ $r[8] }++; 
    # unique code here
    #...
}

That would have eliminated the temporary variable. And so eliminated the result of misspelling a temporary variable. However, USUW works better for these things: USUW="use strict; use warnings;".

Issue regarding string comparisons and references in perl

4 Answers4