Cyrillic symbols shown strangеly when writing to a file

Question

I have a class that has a string field input which contains UTF-8 characters. My class also has a method toString. I want to save instances of the class to a file using the method toString. The problem is that strange symbols are being written in the file:

my $dest = "output.txt";

print "\nBefore saving to file\n" . $message->toString() . "\n";

open (my $fh, '>>:encoding(UTF-8)', $dest) 
    or die "Cannot open $dest : $!";

lock($fh);
print $fh $message->toString();
unlock($fh);
close $fh;       

The first print works fine

Input: {"paramkey":"message","paramvalue":"здравейте"}

is being printed to the console. The problem is when I write to the file:

Input: {"paramkey":"message","paramvalue":"здÑавейÑе"}

I used flock for locking/unlocking the file.

Answer 1

I suppose you miss
use utf8;
in your code...

This code produces the "output.txt" file you do expect:

#!/usr/bin/perl
use strict;
use utf8;

my $dest = "output.txt";
my $message = "здравейте";

print "\nBefore saving to file\n" . $message . "\n";

open (my $fh, '>>:encoding(UTF-8)', $dest)
    or die "Cannot open $dest : $!";

lock($fh);
print $fh $message;
close $fh;

I did not use toString() method because I'm working on native strings, not real objects, but this does not change the substance...

Answer 2

The contents of the string returned by your toString method are already UTF-8 encoded. That works fine when you print it to your terminal because it is expecting UTF-8 data. But when you open your output file with

open (my $fh, '>>:encoding(UTF-8)', $dest) or die "Cannot open $dest : $!"

you are asking that Perl should reencode the data as UTF-8. That converts each byte of the UTF-8-encoded data to a separate UTF-8 sequence, which isn't what you want at all. Unfortunately you don't show your code for the class that $message belongs to, so I can't help you with this

You can fix that by changing your open call to just

open (my $fh, '>>', $dest) or die "Cannot open $dest : $!"

which will avoid the additional encoding step. But you should really be working with unencoded characters throughout your Perl code: removing any encoding from files you are reading from, and encoding output data as necessary when you write to output files.

Answer 3

How does your toString method work? I would guess, based on the output you've provided, that the toString method is producing bytes instead of characters, and then perl is getting confused when trying to convert it.

Try binmode STDOUT, ':encoding(UTF-8)' before your print to see if it produces the same output as the file - otherwise your test is apples and oranges.

If it's already bytes instead of characters, you can open your $dest without any encoding(...) layer and it'll work.

In general, I find it quite painful to work in characters over bytes, but since it resolves more corner cases that I don't have to think about anymore, the extra work becomes worth it, but it is extra work.