Perl Text::Unaccent has unexpected results

Question

I'm experiencing some weird, system dependent issues with the Text::Unaccent module. Apologies if I'm missing something silly, but I've been banging my head against this one for hours with no real progress.

I have a simple script set up that shows the problem reasonably well.

#!/usr/bin/perl
use utf8;
use strict;
use warnings;
use Text::Unaccent;

my $string = 'aaâaa';
my $unacd = unac_string("UTF-8", $string);

print "Accented: $string \n";
print "Unaccented: $unacd \n";

The output on my production server looks great:

[user@prod]$ perl test_unaccent.pl 
Accented: aaâaa
Unaccented: aaaaa

The output on my development server looks strange:

[user@dev]$ perl test_unaccent.pl 
Accented: aaâaa
Unaccented: UTF-8

It just prints out the charset I pass to the unac_string call.

I've checked the locale settings, tried ensuring iconv is working properly (unac_string_utf16 seems to work), but I just can't figure out what could be the problem.

The dev and prod servers are definitely different in a few key ways, but I can't see how it's relevant.

prod: CentOS 5, Perl 5.8.8

dev: CentOS 6, Perl 5.10.1

Thanks in advance for any suggestions/thoughts!

Which version of the Text::Unaccent module do you have installed on each machine? — Jonathan Hall, Nov 16 '13 at 13:15
Both dev and prod have version 1.8.0 (I found this by calling/printing unac_version) — user2999206, Nov 16 '13 at 15:21
That's the version of the underlying library. What about `$Text::Unaccent::VERSION`? — ikegami, Nov 16 '13 at 16:14
woops, yeah, those will always be the same. The only underlying library is `iconv`. Might be the result of different versions of those. — ikegami, Nov 16 '13 at 16:26
Seems to be [this bug](https://rt.cpan.org/Public/Bug/Display.html?id=70995) (got: "ISO-8859-1", expected: "ete"). It looks like the Fedora/CentOS package isn't built correctly. **Edit:** Looking at the [other bugs](https://rt.cpan.org/Public/Dist/Display.html?Name=Text-Unaccent) of that module, it seems to be 64-bit related. — nwellnhof, Nov 16 '13 at 16:39

ikegami · Accepted Answer · 2013-11-16T18:49:12.153

4

I don't know why you're getting the output you are, but what I'm about to say makes it moot.

Your $string isn't a string of UTF-8 bytes; it's a string of Unicode code points. The proper usage^[1] would be

 unac_string("UTF-8", encode("UTF-8", $string));

but since unac_string to converts the string to UTF-16be (using iconv), then passes the string to unac_string_utf16, you might as well use

 unac_string_utf16(encode("UTF-16be", $string));

You could always create a wrapper

use Encode         qw( encode );
use Text::Unaccent qw( unac_string_utf16 );

sub unac_string {
   my ($enc, $str) = @_;
   return unac_string_utf16(encode("UTF-16be", $string));
}

A bug in Text::Unaccent ("The Unicode Bug") makes unac_string("UTF-8", $unicode) usually give the right result, but that same bug makes it so both unac_string("UTF-8", $unicode) and unac_string("UTF-8", $utf8) can give the wrong result. unac_string("UTF-8", encode("UTF-8", $unicode)) will always give the right result, though.

edited Nov 16 '13 at 18:49

answered Nov 16 '13 at 16:28

ikegami

367,544
15
269
518

That works! But... this is a codebase that uses unac_string in a bunch of places. I'd rather not rewrite all of the code, and honestly I'm kind of curious what's going on. Will accept your answer if no one else has any ideas. – user2999206 Nov 16 '13 at 16:36
1

huh? I specifically wrote it so you would only have to make a change in one place! On an unrelated note, I've updated my answer. – ikegami Nov 16 '13 at 16:40
If this were in one file I'd have to change it in one place, but since unac_string is used in several modules across a large codebase, the wrapper would have to be written into all of the modules, or at least it would have to be written into a separate module and included from each place that uses it. I was hoping unac_string could just be fixed. That's not to say I'm not grateful though! Thanks for your assistance! – user2999206 Nov 16 '13 at 19:36
1

so put it in a module, then all you have to do change `use Text::Unaccent;` to `use MyModule;`. Fixing `unac_string` (as oppose to using my shim) won't help the fact that your code is buggy anyway! – ikegami Nov 16 '13 at 23:14
You'd be right, except "unac_string("UTF-8", encode("UTF-8", $string));" does not work. – user2999206 Nov 18 '13 at 12:46

Perl Text::Unaccent has unexpected results

1 Answers1