ansible distribute elements from one list of dicts to different files

Question

I have one list of dicts ( variable ):

sites:
  - { site: "https://site1.com", project: "own", module: "2xx" }
  - { site: "https://site2.com", project: "external", module: "2xx" }
  - { site: "https://site3.net", project: "own", module: "ssl" }
  - { site: "https://site4.org", project: "external", module: "ssl2" }
  - { site: "https://site5.co", project: "external", module: "2xx"}

simple template:

{% for dict_item in sites %}
 - targets:
   - {{ dict_item.site }}
  labels:
   instance: {{ dict_item.site }}
   job: "blackbox_exporter | sites_check"
   project: {{ dict_item.project }}
   module: {{ dict_item.module }}
{% endfor %}

and task :

- name: "Test multi"
  template:
    src: "test.yml.j2"
    dest: "/opt/test/{{ item.module }}.yml"
  with_items: "{{ sites }}"

So, with that task I'm trying to generate separate files ( {{ module}}.yml ) with own set of targets, based on "module" key. But unfortunately inside each of file I have whole list of targets.

What I have:

[root@localhost test]# cat 2xx.yml
 - targets:
   - https://site1.com
  labels:
   instance: https://site1.com
   job: "blackbox_exporter | sites_check"
   project: own
   module: 2xx
 - targets:
   - https://site2.com
  labels:
   instance: https://site2.com
   job: "blackbox_exporter | sites_check"
   project: external
   module: 2xx
 - targets:
   - https://site3.net
  labels:
   instance: https://site3.net
   job: "blackbox_exporter | sites_check"
   project: own
   module: ssl
 - targets:
   - https://site4.org
  labels:
   instance: https://site4.org
   job: "blackbox_exporter | sites_check"
   project: external
   module: ssl2
 - targets:
   - https://site5.co
  labels:
   instance: https://site5.co
   job: "blackbox_exporter | sites_check"
   project: external
   module: 2xx

[root@localhost test]# diff -u 2xx.yml ssl.yml
[root@localhost test]#
[root@localhost test]# diff -u 2xx.yml ssl2.yml
[root@localhost test]#

So content within every file the same and has whole list of records. BUT I want to split it in this way:

[root@localhost test]# cat 2xx.yml
 - targets:
   - https://site1.com
  labels:
   instance: https://site1.com
   job: "blackbox_exporter | sites_check"
   project: own
   module: 2xx
 - targets:
   - https://site2.com
  labels:
   instance: https://site2.com
   job: "blackbox_exporter | sites_check"
   project: external
   module: 2xx
 - targets:
   - https://site5.co
  labels:
   instance: https://site5.co
   job: "blackbox_exporter | sites_check"
   project: external
   module: 2xx

[root@localhost test]# cat ssl.yml
 - targets:
   - https://site3.net
  labels:
   instance: https://site3.net
   job: "blackbox_exporter | sites_check"
   project: own
   module: ssl

[root@localhost test]# cat ssl2.yml
 - targets:
   - https://site4.org
  labels:
   instance: https://site4.org
   job: "blackbox_exporter | sites_check"
   project: external
   module: ssl2

How I can differentiate targets by files ?

How I can differentiate targets by files ? I know that I can add different tasks for each of "module key" and filter it by "when" construction, but in the future I could have lots of modules and save DRY inside my roles tasks

The module `2xx` repeats three times in the *sites*. Do you really want to overwrite the file `/opt/test/{{ item.module }}.yml` three times? — Vladimir Botka, Mar 02 '23 at 01:32
Hello. Thanks for the answer! Your resolution works, but I want to save inside one file several records with the same "module" label ( group records from the list to separate files by the "module" label ). — Nikolai, Mar 02 '23 at 09:53
Good to know. Provide the expected results of these files. [edit] your question and make it [mre]. Your question will be closed if you don't provide a minimal, repeatable example. — Vladimir Botka, Mar 02 '23 at 09:59
I updated the answer. Fit the structure to your needs. Do you really need the lists? Wouldn't dictionaries do the job? — Vladimir Botka, Mar 02 '23 at 16:10

Vladimir Botka · Accepted Answer · 2023-03-02T16:09:29.867

Q: "Generate separate files {{ module }}.yml"

A: For example, given the simplified list

  sites:
    - {site: https://site1.com, project: own, module: 2xx}
    - {site: https://site2.com, project: external, module: 2xx}
    - {site: https://site3.net, project: own, module: ssl}
    - {site: https://site4.org, project: external, module: ssl2}
    - {site: https://site5.co, project: external, module: 2xx}

Write the files

    - copy:
        dest: "/tmp/test/{{ item.0 }}.yml"
        content: |
          {% for s in item.1 %}
          {{ s.site }}:
            instance: {{ s.site }}
            project: {{ s.project }}
            module: {{ s.module }}
          {% endfor %}
      loop: "{{ sites|groupby('module') }}"
      loop_control:
        label: "{{ item.0 }}"

This will create the files

shell> tree /tmp/test
/tmp/test
├── 2xx.yml
├── ssl2.yml
└── ssl.yml

0 directories, 3 files

shell> cat /tmp/test/2xx.yml 
https://site1.com:
  instance: https://site1.com
  project: own
  module: 2xx
https://site2.com:
  instance: https://site2.com
  project: external
  module: 2xx
https://site5.co:
  instance: https://site5.co
  project: external
  module: 2xx

shell> cat /tmp/test/ssl2.yml 
https://site4.org:
  instance: https://site4.org
  project: external
  module: ssl2

shell> cat /tmp/test/ssl.yml 
https://site3.net:
  instance: https://site3.net
  project: own
  module: SSL

Use the module assemble if you want to put them together

    - assemble:
        src: /tmp/test
        dest: /tmp/test/all.yml

This will create the file

shell> cat /tmp/test/all.yml 
https://site1.com:
  instance: https://site1.com
  project: own
  module: 2xx
https://site2.com:
  instance: https://site2.com
  project: external
  module: 2xx
https://site5.co:
  instance: https://site5.co
  project: external
  module: 2xx
https://site3.net:
  instance: https://site3.net
  project: own
  module: ssl
https://site4.org:
  instance: https://site4.org
  project: external
  module: ssl2

You can read it and put it into a dictionary

    - include_vars:
        file: /tmp/test/all.yml
        name: targets

gives

  targets:
    https://site1.com:
      instance: https://site1.com
      module: 2xx
      project: own
    https://site2.com:
      instance: https://site2.com
      module: 2xx
      project: external
    https://site3.net:
      instance: https://site3.net
      module: ssl
      project: own
    https://site4.org:
      instance: https://site4.org
      module: ssl2
      project: external
    https://site5.co:
      instance: https://site5.co
      module: 2xx
      project: external

^{Example of a complete playbook for testing
- hosts: localhost

vars:

sites:
- {site: https://site1.com, project: own, module: 2xx}
- {site: https://site2.com, project: external, module: 2xx}
- {site: https://site3.net, project: own, module: ssl}
- {site: https://site4.org, project: external, module: ssl2}
- {site: https://site5.co, project: external, module: 2xx}

tasks:

- debug:
msg: "{{ sites|groupby('module') }}"
when: debug|d(false)|bool

- copy:
dest: "/tmp/test/{{ item.0 }}.yml"
content: |
{% for s in item.1 %}
{{ s.site }}:
instance: {{ s.site }}
project: {{ s.project }}
module: {{ s.module }}
{% endfor %}
loop: "{{ sites|groupby('module') }}"
loop_control:
label: "{{ item.0 }}"

- assemble:
src: /tmp/test
dest: /tmp/test/all.yml

- include_vars:
file: /tmp/test/all.yml
name: targets

- debug:
var: targets}

ansible distribute elements from one list of dicts to different files

1 Answers1