11

I am in the file encoding hell with Puppet. Even the simplest try does not work:

hiera-data/test.yaml:
---
test: Äñö

init.pp:
  $test = hiera('test')
  file { "/root/encoding.txt":
    ensure  => file,
    content => $test
  }

On the Puppet server everything looks fine:

puppet:~ # file -i /etc/puppetlabs/puppet/hiera-data/env/test.yaml
/etc/puppetlabs/puppet/hiera-data/env/test.yaml: text/plain charset=utf-8
puppet:~ # cat /etc/puppetlabs/puppet/hiera-data/env/test.yaml
---
test: Äñö
puppet:~ # locale
LANG=POSIX
LC_CTYPE=en_US.UTF-8

On the puppet agent:

puppet-test:~ # locale
LANG=POSIX
LC_CTYPE=en_US.UTF-8

After running:

puppet-test:~ # file -i encoding.txt
encoding.txt: text/plain charset=utf-8

but

cat encoding.txt

Here is the HEX data asked for:

0000000: efbf bdef bfbd efbf bdef bfbd efbf bdef  ................
0000010: bfbd 0a                                  ...

Running hiera directly does not provide any further insight. In special, I can only try it on the server, since the agent does not have the sources. enter image description here

My environment is quite outdated, but I am not allowed to use any newer version, at least not yet, without any GOOD reason:

  • SuSE Enterprise Linux 11 Service Pack 3
  • Puppet Enterprise 3.8.6
  • pe-ruby-1.9.3.551-9.pe.sles11

I would appreciate any insight to this problem.

the Tin Man
  • 158,662
  • 42
  • 215
  • 303
mmoossen
  • 1,237
  • 3
  • 21
  • 32
  • 6
    Are you sure that the problem is not your terminal? Open `encoding.txt` in a hex editor (or in a text editor with a binary / hex mode) and check whether the file contains the correct bytes. In particular, the UTF-8 encoding for the string you present would consist of these six bytes: `c3 84 c3 b1 c3 b6`. If it contains something different, then please add that to your question. – John Bollinger May 05 '17 at 13:35
  • i updated the question with the missing info, Thanks @JohnBollinger – mmoossen May 06 '17 at 14:32
  • After a second look at this, this does not seem like a puppet problem. This seems like a system env problem. – Matthew Schuchard May 10 '17 at 21:48
  • @MattSchuchard: could be, but what could i check to get further? – mmoossen May 11 '17 at 07:04
  • 1
    What do you get if you use hiera from the command line? e.g.: ` # hiera -d test > test-encoding.txt ` (you may have to use -c to find the right files) – Peter Faller May 12 '17 at 14:02
  • Question updated, thanks @PeterFaller, but no new insight... – mmoossen May 16 '17 at 14:54
  • What's the locale on the server ? The bytes in `encoding.txt` don't seem to make any sense. Here's how to recreate this file for debugging: `File.open('encoding.txt','w') { |f| f.write ["efbfbdefbfbdefbfbdefbfbdefbfbdefbfbd0a"].pack('H*') }` – Fravadona Oct 23 '21 at 11:56

1 Answers1

0

Having LANG=POSIX can definitely make things screwy. LANG is what determines the encoding that the console displays. It is usually desirable to have LANG and LC_TYPE matching.

See: "Explain the effects of export LANG, LC_CTYPE, LC_ALL".

And for more info on LANG=POSIX see POSIX Locale.

Also, on an unrelated note, Puppet 3.8 definitely has defects around properly displaying and/or persisting Unicode characters. A lot of work has been done in the more recent Puppet builds to completely internationalize and localize Puppet.

the Tin Man
  • 158,662
  • 42
  • 215
  • 303
Sam Woods
  • 1,820
  • 14
  • 13
  • Hi @Sam, sorry but your answer, nor the links, give me any glue what to try next... anyhow, i get the same results on a node using `LANG=de_DE.UTF-8`. But still many thanks for answering... – mmoossen Sep 30 '17 at 19:28