12

I'm using Thor and trying to output YAML to a file. In irb I get what I expect. Plain text in YAML format. But when part of a method in Thor, its output is different...

class Foo < Thor
  include Thor::Actions

  desc "bar", "test"
  def set
    test = {"name" => "Xavier", "age" => 30}
    puts test
    # {"name"=>"Xavier", "age"=>30}
    puts test.to_yaml
    # !binary "bmFtZQ==": !binary |-
    #   WGF2aWVy
    # !binary "YWdl": 30
    File.open("data/config.yml", "w") {|f| f.write(test.to_yaml) }
  end
end

Any ideas?

cp3
  • 2,119
  • 2
  • 22
  • 27
  • 1
    I just ran your example and it gave me perfectly fine output. I ran thor 0.14.6. – Maran Mar 03 '12 at 23:14
  • Thanks for taking the time to check. I'm at a loss of what to do at this point. I'm using Ruby 1.9.3p125 if that makes any difference what so ever. :) – cp3 Mar 04 '12 at 02:00
  • 1
    I installed 1.9.3 and ran it again, and indeed binary output. I noticed YAML got upgraded during install. It probably has something to do with that upgraded version. – Maran Mar 04 '12 at 08:05
  • Thanks Maran. I assume yaml is built into ruby, would it be easier to rvm install 1.9.2 rather than install 1.9.3 w/ the downgraded version of yaml? – cp3 Mar 04 '12 at 09:14
  • Yeah, as soon as I issue rvm 1.9.2 everything works again, yaml wise. If you don't _need_ 1.9.3 I would switch back to 1.9.2 – Maran Mar 04 '12 at 12:22
  • I tried it using Ruby 1.9.3p0 and Thor 0.14.6 and it generates perfect yaml for me. Could it be the difference between p0 and p125? – Paul Simpson Mar 24 '12 at 05:31
  • Adding `#encoding: UTF-8` to the top of the file seems to fix it. I don't know what's going on though - it looks like thor is changing the default encoding to ASCII-8BIT and then [a change to yaml in Ruby 1.9.3](https://github.com/tenderlove/psych/commit/c9cd187d5aa8fa6607dd463b5f98a65483ae39ce) causes the output to be binary. – matt Mar 29 '12 at 22:11
  • I think it's due to how thor reads in `.thor` files; it uses [`File.binread`](http://ruby-doc.org/core-1.9.3/IO.html#method-c-binread) which sets the encoding to ASCII-8BIT. – matt Mar 29 '12 at 22:19

3 Answers3

14

All Ruby 1.9 strings have an encoding attached to them.

YAML encodes some non-UTF8 strings as binary, even when they look innocent, without any high-bit characters. You might think that your code is always using UTF8, but builtins can return non-UTF8 strings (ex File path routines).

To avoid binary encoding, make sure all your strings encodings are UTF-8 before calling to_yaml. Change the encoding with force_encoding("UTF-8") method.

For example, this is how I encode my options hash into yaml:

options = {
    :port => 26000,
    :rackup => File.expand_path(File.join(File.dirname(__FILE__), "../sveg.rb"))
}
utf8_options = {}
options.each_pair { |k,v| utf8_options[k] = ((v.is_a? String) ? v.force_encoding("UTF-8") : v)}
puts utf8_options.to_yaml

Here is an example of yaml encoding simple strings as binary

>> x = "test"
=> "test"
>> x.encoding
=> #<Encoding:UTF-8>
>> x.to_yaml
=> "--- test\n...\n"
>> x.force_encoding "ASCII-8BIT"
=> "test"
>> x.to_yaml
=> "--- !binary |-\n  dGVzdA==\n"
Aleksandar Totic
  • 2,557
  • 25
  • 27
7

After version 1.9.3p125, ruby build-in YAML engine will treat all BINARY encoding differently than before. All you need to do is to set correct non-BINARY encoding before your String.to_yaml.

in Ruby 1.9, All String object have attached a Encoding object and as following blog ( by James Edward Gray II ) mentioned, ruby have build in three type of encoding when String is generated: http://blog.grayproductions.net/articles/ruby_19s_three_default_encodings.

One of encoding may solve your problem => Source code Encoding

This is the encoding of your source code, and can be specify by adding magic encoding string at the first line or second line ( if you have a sha-bang string at the first line of your source code ) the magic encoding code could be one of following:

  • # encoding: utf-8
  • # coding: utf-8
  • # -- encoding : utf-8 --

so in your case, if you use ruby 1.9.3p125 or later, this should be solved by adding one of magic encoding in the beginning of your code.

# encoding: utf-8
require 'thor'
class Foo < Thor
  include Thor::Actions

  desc "bar", "test"
  def bar
    test = {"name" => "Xavier", "age" => 30}
    puts test
    #{"name"=>"Xavier", "age"=>30}
    puts test["name"].encoding.name
    #UTF-8
    puts test.to_yaml
    #---
    #name: Xavier
    #age: 30
    puts test.to_yaml.encoding.name
    #UTF-8
  end
end
Simon Iong
  • 191
  • 3
  • 9
0

I have been struggling with this using 1.9.3p545 on Windows - just with a simple hash containing strings - and no Thor.

The gem ZAML solves the problem quite simply:

require 'ZAML'
yaml = ZAML.dump(some_hash)
File.write(path_to_yaml_file, yaml)
Ross Attrill
  • 2,594
  • 1
  • 22
  • 31