25

I have a list of dictionaries, which I want to serialize:

list_of_dicts = [ { 'key_1': 'value_a', 'key_2': 'value_b'},
                  { 'key_1': 'value_c', 'key_2': 'value_d'},
                  ...
                  { 'key_1': 'value_x', 'key_2': 'value_y'}  ]

yaml.dump(list_of_dicts, file, default_flow_style = False)

produces the following:

- key_1: value_a
  key_2: value_b
- key_1: value_c
  key_2: value_d
(...)
- key_1: value_x
  key_2: value_y

But i'd like to get this:

- key_1: value_a
  key_2: value_b
                     <-|
- key_1: value_c       | 
  key_2: value_d       |  empty lines between blocks
(...)                  |
                     <-|
- key_1: value_x
  key_2: value_y

PyYAML documentation talks about dump() arguments very briefly and doesn't seem to have anything on this particular subject.

Editing the file manually to add newlines improves readability quite a lot, and the structure still loads just fine afterwards, but I have no idea how to make dump method generate it.

And in general, is there a way to have more control over output formatting besides simple indentation?

Anthon
  • 69,918
  • 32
  • 186
  • 246
nope
  • 387
  • 1
  • 3
  • 8

3 Answers3

21

There's no easy way to do this with the library (Node objects in yaml dumper syntax tree are passive and can't emit this info), so I ended up with

stream = yaml.dump(list_of_dicts, default_flow_style = False)
file.write(stream.replace('\n- ', '\n\n- '))
Yuri Baburov
  • 712
  • 7
  • 17
  • 3
    Thanks! Had to use something similar for lists formatting. PyYAML does not put indent before `-`, while YAML consuming library which we use expected some indentation there. So we had to do `replace('- ', ' - ')` – Andrei Jul 07 '14 at 14:48
  • That node are passive is true but irrelevant, as nodes don't emit any other information either (i.e. `ScalarNode`s don't emit their own value). The `Emitter` does get a node's value (when appropriate) and emits it, and if you attach extra information on nodes, and enhance the relevant Emitter methods to handle this extra information (like I do in `ruamel.yaml`), then there is absolutely no need to do that kind of, coarse, string based post-processing. – Anthon Aug 15 '17 at 14:25
  • @Andrei With [`ruamel.yaml`](https://pypi.python.org/pypi/ruamel.yaml) you can set `yaml.indent(sequence=3, offset=1)` and get that output without post-processing. – Anthon Aug 15 '17 at 14:27
8

PyYAML documentation only talks about dump() arguments briefly, because there is not much to say. This kind of control is not provided by PyYAML.

To allow preservation of such empty (and comment) lines in YAML that is loaded, I started the development of the ruamel.yaml library, a superset of the stalled PyYAML, with YAML 1.2 compatibility, many features added and bugs fixed. With ruamel.yaml you can do:

import sys
import ruamel.yaml

yaml_str = """\
- key_1: value_a
  key_2: value_b

- key_1: value_c
  key_2: value_d

- key_1: value_x  # a few before this were ellipsed
  key_2: value_y
"""

yaml = ruamel.yaml.YAML()
data = yaml.load(yaml_str)
yaml.dump(data, sys.stdout)

and get the output exactly the same as the input string (including the comment).

You can also build the output that you want from scratch:

import sys
import ruamel.yaml

yaml = ruamel.yaml.YAML()
list_of_dicts = yaml.seq([ { 'key_1': 'value_a', 'key_2': 'value_b'},
                           { 'key_1': 'value_c', 'key_2': 'value_d'},
                           { 'key_1': 'value_x', 'key_2': 'value_y'}  ])

for idx in range(1, len(list_of_dicts)):
    list_of_dicts.yaml_set_comment_before_after_key(idx, before='\n')

ruamel.yaml.comments.dump_comments(list_of_dicts)
yaml.dump(list_of_dicts, sys.stdout)

The conversion using yaml.seq() is necessary to create an object that allows attachment of the empty-lines through special attributes.

The library also allows preservation/easy-setting of quotes and literal style on strings, format of int (hex, octal, binary) and floats. As well as separate indent specification for mappings and sequences (although not for individual mappings or sequences).

Anthon
  • 69,918
  • 32
  • 186
  • 246
  • 2
    Good answer, and cheers for forking your own improved yaml parser, I'll have a go with that now as I'm looking into dump formatting issues myself now. It occurs to me though, pyyaml has continued to be updated since this post. Have they improved on formatting since? Or have any of your contributions been merged into pyyaml? – Iron Attorney Jun 08 '19 at 14:41
  • 1
    No on both questions. The only real chance that I know of is that PyYAML's `load()` is no longer as dangerous if there is no 100% control over the input. It is still YAML 1.1, multiple long-standing bugs are still not fixed. Changes seem to be mostly for compatibility with newer versions of Python. I think my (and all other PRs) were dropped with the move from bitbucket to github (an effort that would have been better spent on bugfixing PyYAML IMHO). ruamel.yaml now has options to control an extra newline after the item indicator (`-`): `.compact_seq_seq` resp. `.compact_seq_map` – Anthon Jun 08 '19 at 15:08
4

While its a little klunky, I had the same goal as OP. I solved it by subclassing yaml.Dumper

from yaml import Dumper

class MyDumper(Dumper):

  def write_indent(self):
    indent = self.indent or 0
    if not self.indention or self.column > indent \
        or (self.column == indent and not self.whitespace):
      self.write_line_break()


    ##########$#######################################
    # On the first level of indentation, add an extra
    # newline

    if indent == 2:
      self.write_line_break()

    ##################################################

    if self.column < indent:
      self.whitespace = True
      data = u' '*(indent-self.column)
      self.column = indent
      if self.encoding:
        data = data.encode(self.encoding)
      self.stream.write(data)

You call it like this:

print dump(python_dict, default_flow_style=False, width=79, Dumper=MyDumper)
synthesizerpatel
  • 27,321
  • 5
  • 74
  • 91