How to load only the unique items of per file stored lists identified via the same key path into a single variable using ansible?

Question

Problem

How to load only the unique items of per file stored lists identified via the same key path into a single variable using ansible?

Scenario

There is one specific folder /myfolder on a remote machine that includes files with pattern based name file-*.yml, while they have the same structure (key path to list is always the same - same-root-key.same-list-key). Contents are shown below:

• file file-1.yml with the following content:

# /myfolder/file-1.yml
---
same-root-key:
  same-list-key:
    - a
    - b
    - c

• file file-2.yml with the following content:

# /myfolder/file-2.yml
---
same-root-key:
  same-list-key:
    - a
    - x

• file file-3.yml with the following content:

# /myfolder/file-3.yml
---
same-root-key:
  same-list-key:
    - b

• file file-4.yml with the following content:

# /myfolder/file-4.yml
---
same-root-key:
  same-list-key:
    - y

• file file-5.yml with the following content:

# /myfolder/file-5.yml
---
same-root-key:
  same-list-key: []

Expectation

- name: load only the unique items of per file stored lists
  set_fact:
    result: # somehow load the expected items

- debug:
    var: result

#  "result": [
#    "a",
#    "b",
#    "c",
#    "x",
#    "y"
#  ]

Idea to concept (WORKING)

main.yml content:

# get list of file paths from remote machine
- find:
    paths: "/myfolder"
    patterns: "file-*.yml"
  register: folder_files

# iteration
- name: concept
  include_tasks: ./block-file.yml
  loop: "{{ folder_files.files }}"

- debug:
    var: results

block-file.yml content:

---
# load remote file content
- slurp:
    src: '{{ item.path }}'
  register: tmp_file

# get list values
- set_fact:
    tmp_fact: "{{ tmp_file.content | b64decode | from_yaml }}"

# lists union
- set_fact:
    result: "{{ result|default([]) | union(tmp_fact.same-root-key.same-list-key) }}"

Related

• In Ansible, how to combine variables from separate files into one array?
  • with_fileglob works for local files (not for files on remote machine)
• Ansible: read remote file
  • lookup() works for local files (not for files on remote machine)
• https://docs.ansible.com/ansible/latest/collections/ansible/builtin/slurp_module.html
  • slurp should be used to load remote file content
• Is there with_fileglob that works remotely in ansible?
  • with_fileglob on remote machine can be done using find
• https://serverfault.com/questions/737007/how-to-combine-two-lists
  • union should be used to create a set (list consisting of unique values) from two lists
• Issue looping on block module for a set of tasks in Ansible
  • our case may require an iteration over a block, which is not supported as a direct loop over a block
  • the block must be stored in a separate file, while looping over a file using include_tasks is supported

Notes

Later, after I have posted this topic, I have fixed the concept and now it's working. But I am still curious how can I achieve the same behavior in a much more elegant and cleaner way. The above mentioned concept was updated.

Answer 1

A simpler approach would be to fetch the files, e.g.

    - fetch:
        dest: "{{ fetch_dir }}"
        src: "{{ item.path }}"
      loop: "{{ folder_files.files }}"

given the remote host is "test_11" and "fetch_dir=fetch" gives

shell> tree fetch/test_11/myfolder/
fetch/test_11/myfolder/
├── file-1.yml
├── file-2.yml
├── file-3.yml
├── file-4.yml
└── file-5.yml

Then, collect the lists, e.g.

    - set_fact:
        tmp_fact: "{{ tmp_fact|default([]) +
                      (lookup('file', fetch_dir ~ '/' ~
                                      inventory_hostname ~ '/' ~
                                      item.path)|
                       from_yaml)['same-root-key']['same-list-key'] }}"
      loop: "{{ folder_files.files }}"

gives

  tmp_fact:
  - a
  - b
  - c
  - y
  - b
  - a
  - x

Then, select the unique items

    - set_fact:
        result: "{{ tmp_fact|unique }}"

gives

  result:
  - a
  - b
  - c
  - y
  - x

---

Q: "The solution ... downloads files ... in contrast to the solution ... in the question (section: Idea to concept)"

A: You might want to reconsider the concept. fetch is more user-friendly compared to slurp. The proposed solution is idempotent, i.e. the files will be downloaded only when changed. Running the task repeatedly

    - fetch:
        dest: "{{ fetch_dir }}"
        src: "{{ item.path }}"
      loop: "{{ folder_files.files }}"
      loop_control:
        label: "{{ item.path }}"

gives

TASK [fetch] *************************************************************
ok: [test_11] => (item=/myfolder/file-1.yml)
ok: [test_11] => (item=/myfolder/file-4.yml)
ok: [test_11] => (item=/myfolder/file-3.yml)
ok: [test_11] => (item=/myfolder/file-2.yml)
ok: [test_11] => (item=/myfolder/file-5.yml)

As a result, the playbook

- hosts: test_11
  gather_facts: false
  vars:
    fetch_dir: fetch
  tasks:
    - find:
        paths: "/myfolder"
        patterns: "file-*.yml"
      register: folder_files
    - fetch:
        dest: "{{ fetch_dir }}"
        src: "{{ item.path }}"
      loop: "{{ folder_files.files }}"
      loop_control:
        label: "{{ item.path }}"

    - set_fact:
        tmp_fact: "{{ tmp_fact|default([]) +
                      (lookup('file', fetch_dir ~ '/' ~
                                      inventory_hostname ~ '/' ~
                                      item.path)|
                       from_yaml)['same-root-key']['same-list-key'] }}"
      loop: "{{ folder_files.files }}"

shows no changes when running repeatedly

PLAY RECAP **************************************************************
test_11: ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

In your concept, you use slurp. This module always transfers the data from the remote host. The fetch module is more efficient. It compares the checksums and transfers the data only if the files are different. In addition to this, the fetch module supports check_mode.