Ansible Loops in query Module

Question

Hi I have a requirement to read the content of all the files of specific directory and pass the content to a variable.

- name: Generate  logs
  hosts: appserver
  vars:
    logs_dir: /properties
    logs: {}
  tasks:
    - name: Read log config files
      set_fact:
        log_files: "{{ lookup('fileglob', logs_dir + '/*.properties')|split(',') }}"
    - name: check values   
      debug:
        var: log_files
      
    - name: Generate  list
      set_fact:
         logs: "{{ item }}"
      with_items: "{{ query('file', log_files[1], wantlist=True) }}"
       

It is working when I am passing only one file I want to pass file by file to query module ? any suggestion and these are yaml files and please do let me help in fetching specific tags from file as well.

Answer 1

Q: "Read all (yaml) files of specific directory and pass the content to a variable."

A: There are more options. For example, given the tree for testing

shell> tree /tmp/properties
/tmp/properties
├── 01.properties
├── 02.properties
└── 03.properties

0 directories, 3 files

shell> cat /tmp/properties/01.properties 
var_01: '01'

shell> cat /tmp/properties/02.properties 
var_02: '02'

shell> cat /tmp/properties/03.properties 
var_03: '03'

---

1. Include the variables from all files and combine a dictionary. The play

- hosts: localhost

  vars:

    log_dir: /tmp/properties
    log_files_glob: "{{ log_dir }}/*.properties"
    log_files: "{{ query('fileglob', log_files_glob) }}"
    log_files_dict: "{{ lookup('vars', *q('varnames', 'log_file_\\d+'))|combine }}"

  tasks:

    - block:
        - debug:
            var: log_files
        - include_vars:
            file: "{{ item }}"
            name: "log_file_{{ ansible_loop.index }}"
          loop: "{{ log_files }}"
          loop_control:
            extended: true
        - debug:
            var: log_files_dict
      run_once: true

gives (abridged)

  log_files_dict:
    var_01: '01'
    var_02: '02'
    var_03: '03'

---

2. You can simplify the code and read the standalone variables

        - include_vars: "{{ item }}"
          loop: "{{ log_files }}"
        - debug:
            msg: |
              var_01: {{ var_01 }}
              var_02: {{ var_02 }}
              var_03: {{ var_03 }}

gives

  msg: |-
    var_01: 01
    var_02: 02
    var_03: 03

---

3. The next option is using the module assemble

        - assemble:
            src: "{{ log_dir }}"
            regexp: '.*\.properties'
            dest: /tmp/properties.yml

gives the file

shell> cat /tmp/properties.yml
var_01: '01'
var_02: '02'
var_03: '03'

Include this file as you like.

---

Example of a complete playbook for testing

- hosts: localhost

  vars:

    log_dir: /tmp/properties

  tasks:

    - block:
        - assemble:
            src: "{{ log_dir }}"
            regexp: '.*\.properties'
            dest: /tmp/properties.yml
        - include_vars:
            file: /tmp/properties.yml
            name: log_files_dict
        - debug:
            var: log_files_dict
      run_once: true

---

Q: "The data comes with backslashes like \"test\":\"somedata\",

A: You'll get escaped quotes \" if you quote a JSON string. For example, the value of the variable var1 below is neither YAML nor JSON. Because of the wrapping single quotes ' it is a string.

shell> cat /tmp/properties/01.properties 
var_01: '{"test": "somedata", "config": {"somedata": "somedata"}}'

The play gives (in JSON)

shell> ANSIBLE_STDOUT_CALLBACK=default ansible-playbook pb.yml
  ...
    "log_files_dict": {
        "var_01": "{\"test\": \"somedata\", \"config\": {\"somedata\": \"somedata\"}}",
        "var_02": "02",
        "var_03": "03"
    }

In YAML, the single quotes ' can be used. As a result, the double quotes " don't have to be escaped

shell> ANSIBLE_STDOUT_CALLBACK=yaml ansible-playbook pb.yml
  ...
  log_files_dict:
    var_01: '{"test": "somedata", "config": {"somedata": "somedata"}}'
    var_02: '02'
    var_03: '03'

When you remove the wrapping quotes the value will be a valid JSON (and valid YAML as well because JSON is a subset of YAML).

shell> cat /tmp/properties/01.properties 
var_01: {"test": "somedata", "config": {"somedata": "somedata"}}

shell> ANSIBLE_STDOUT_CALLBACK=default ansible-playbook pb.yml
  ...
    "log_files_dict": {
        "var_01": {
            "config": {
                "somedata": "somedata"
            },
            "test": "somedata"
        },
        "var_02": "02",
        "var_03": "03"
    }

shell> ANSIBLE_STDOUT_CALLBACK=yaml ansible-playbook pb.yml
  ...
  log_files_dict:
    var_01:
      config:
        somedata: somedata
      test: somedata
    var_02: '02'
    var_03: '03'