How can I print a jinja dict in a deterministic order?

Question

I have a configuration file where a particular setting takes a long ugly json-style dictionary. I would like to define this dictionary in YAML and print it out with Jinja. Currently this looks something like this:

{%- load_yaml as some_conf_val %}
val1: something
val2: completely
val3: different
{%- endload %}

option = {{ some_conf_val }}

# Results in:
option = {'val2': 'completely', 'val1': 'something', 'val3': 'different'}

By happy coincidence this is exactly the format expected by the program being configured, and the yaml block is much easier to read and modify than the inline version. However, the fact that the keys come out in neither alphabetical order nor the order they were defined leads me to suspect their output order is non-deterministic. On re-running the state a few times they always come out in the same order, but that doesn't prove much.

This is a problem because if the output string is altered, it gets treated as a change to the file and triggers a service restart, even if nothing functional has been changed. I don't care what specific order the keys are in, but I do need that order to be the same every time.

How can I accomplish this? Or is it already deterministic and just doesn't look like it?

(If I understand correctly jinja dicts are really python dicts under the hood, and python dicts are unordered, so this may be impossible without including code that's messier than the line I'm trying to not have to write. But I'm hoping not.)

Dauros · Answer 1 · 2015-11-18T19:22:49.757

3

You are right, Python's standard dict is unordered. There is an ordered version in the collections module: collections.OrderedDict. We can use it for this purpose with a custom Jinja2 filter.

The custom filter first converts the input to OrderedDict, then to JSON, and finally replaces the double quotation marks:

from json import dumps
from collections import OrderedDict

def conffilter(value):
    return dumps(OrderedDict(value)).replace('"', '\'')

Then we have to add this new filter to Jinja2:

 env.filters['conffilter'] = conffilter

where env is the Jinja2 Environment object from your app (e.g. in Flask it's the app.jinja_env).

Now we can use the new filter with Jinja2's dictsort filter.

# Define a list for testing:
{%- set some_conf_val = {'val2': 'completely', 'val1': 'something', 'val3': 'different'} -%}

# Unordered dict output
option = {{ some_conf_val }}
# Result:
option = {'val1': 'something', 'val3': 'different', 'val2': 'completely'}

# Ordered list of tuples
option = {{ some_conf_val|dictsort }}
# Result:
option = [('val1', 'something'), ('val2', 'completely'), ('val3', 'different')] 

# Ordered dict-like output:
option = {{ some_conf_val|dictsort|conffilter }}
# Result:
option = {'val1': 'something', 'val2': 'completely', 'val3': 'different'}

edited Nov 18 '15 at 19:22

answered Nov 18 '15 at 07:21

Dauros

9,294
2
20
28

The documentation says dictsort returns (key, value) pairs. Wouldn't this print something like `option = [('val1', 'something'), ('val2', 'completely'), ('val3', 'different')]`? – Andrew Nov 18 '15 at 15:36
True, I missed that detail. I updated my answer adding a custom Jinja2 filter which converts that to a dict-like format. – Dauros Nov 18 '15 at 19:25
1

Unfortunately saltstack doesn't provide a way to register custom filters, so this solution still won't work for me. However, it seems that it *does* provide a json filter, and said filter does sort keys in the output, which solved my problem. I'm going to upvote you anyway because I think your answer is still useful for people trying to do this outside the context of saltstack. – Andrew Nov 18 '15 at 19:35
1

@Andrew for future reference, SaltStack does provide a way to register custom execution modules, which can be accessed in jinja via `salt['my.function']`. – OrangeDog Jun 27 '20 at 23:05

score 3 · Accepted Answer · answered Nov 18 '15 at 19:47

It turns out that saltstack provides a filter for json output that sorts its keys and can be massaged to get what I wanted. The solution I eventually used looks like this:

option = {{ some_conf_val|json|replace('"', "'") }}

# Results in:
option = {'val1': 'something', 'val2': 'completely', 'val3': 'different'}

The replace operation is because the option eventually gets fed to something that cares what kind of quotes it's looking at. It may not apply in other circumstances.

As far as I can tell the sorting behavior isn't documented, but you can find it in the source here.

How can I print a jinja dict in a deterministic order?

2 Answers2