Ansible : multy loop into json file

Question

I have as a source a json file that contains a list of NetworkFlow keys, and from which i would like to extract information to create security rules, using a double loop in ansible.

Below an example from my json file :

{
   "Name":"Some_name",
   "NetworkFlow":[
      {
         "GroupName":"Test1",
         "Type":"Ingress",
         "Env":"prod",
         "Server":[
            "192.168.1.1",
            "192.168.1.2"
         ],
         "Service":[
            {
               "Protocol":"TCP",
               "Port":"443,22,53"
            },
            {
               "Protocol":"UDP",
               "Port":"21"
            }
         ]
      },
      {
         "GroupName":"Test2",
         "Type":"Egress",
         "Env":"dev",
         "Server":[
            "192.168.1.3",
            "192.168.1.4"
         ],
         "Service":[
            {
               "Protocol":"UDP",
               "Port":"9996,9997"
            }
         ]
      }
   ]
}

so firstly i have to loop for each NetworkFlow section, and inside each one, i have to loop in the list of servers and also in the list of services (protocols and ports) to get a simular parsing like the below:

#rule= Server,Protocol,Port,Type,Env,GroupName
  msg: 192.168.1.1,TCP,443,Ingress,prod,Test1
  msg: 192.168.1.1,TCP,22,Ingress,prod,Test1
  msg: 192.168.1.1,TCP,53,Ingress,prod,Test1
  msg: 192.168.1.1,UDP,21,Ingress,prod,Test1
  msg: 192.168.1.2,TCP,443,Ingress,prod,Test1
  msg: 192.168.1.2,TCP,22,Ingress,prod,Test1
  msg: 192.168.1.2,TCP,53,Ingress,prod,Test1
  msg: 192.168.1.2,UDP,21,Ingress,prod,Test1
  
  msg: 192.168.1.3,UDP,9996,Egress,dev,Test2
  msg: 192.168.1.3,UDP,9997,Egress,dev,Test2
  msg: 192.168.1.4,UDP,9996,Egress,dev,Test2
  msg: 192.168.1.4,UDP,9997,Egress,dev,Test2

Here Below my playbook tasks :

my main task :

---
- name: Include JSON file
  include_vars:
    file: test.json

- include_tasks: rules.yml
  loop: "{{ NetworkFlow }}"
  loop_control:
    loop_var: oi

my rule task :

---
- set_fact:
    Services: "{{ Services|from_yaml }}"
  vars:
    Services: |
      {% for service in oi.Service %}
      {% for port in service.Port.split(',') %}
        - Protocol: {{ service.Protocol }}
          Port: {{ port }}
      {% endfor %}
      {% endfor %}

- debug:
    msg: "{{ i.0 }},{{ i.1.Protocol }},{{ i.1.Port }},{{ oi.Type }},{{ oi.Env }},{{ oi.GroupName }}"
  with_nested:
    - "{{ oi.Server }}"
    - "{{ Services }}"
  loop_control:
    loop_var: i

For info, my oi.Service.Port can have a list of port separated by a comma !

I tried with a loop inside a with_nested and it work for the first key Test1, but i didn't get the correct parsing for the second NetworkFlow key Test2

TASK [test : set_fact] *****************************************************************************************************************************************
ok: [localhost]

TASK [test : debug] ********************************************************************************************************************************************
    "msg": "192.168.1.1,TCP,443,Ingress,prod,Test1"
    "msg": "192.168.1.1,TCP,22,Ingress,prod,Test1"
    "msg": "192.168.1.1,TCP,53,Ingress,prod,Test1"
    "msg": "192.168.1.1,UDP,21,Ingress,prod,Test1"
    "msg": "192.168.1.2,TCP,443,Ingress,prod,Test1"
    "msg": "192.168.1.2,TCP,22,Ingress,prod,Test1"
    "msg": "192.168.1.2,TCP,53,Ingress,prod,Test1"
    "msg": "192.168.1.2,UDP,21,Ingress,prod,Test1"
}

TASK [test : set_fact] *****************************************************************************************************************************************
ok: [localhost]

TASK [test : debug] ********************************************************************************************************************************************
    "msg": "192.168.1.3,TCP,443,Egress,dev,Test2"
    "msg": "192.168.1.3,TCP,22,Egress,dev,Test2"
    "msg": "192.168.1.3,TCP,53,Egress,dev,Test2"
    "msg": "192.168.1.3,UDP,21,Egress,dev,Test2"
    "msg": "192.168.1.4,TCP,443,Egress,dev,Test2"
    "msg": "192.168.1.4,TCP,22,Egress,dev,Test2"
    "msg": "192.168.1.4,TCP,53,Egress,dev,Test2"
    "msg": "192.168.1.4,UDP,21,Egress,dev,Test2"

Have anyone idea for how to deal with that please?

Are you aware of the [`| groupby` filter](https://jinja.palletsprojects.com/en/2.11.x/templates/#groupby) -- I'd have to really study your problem to figure out what is wrong, but using a filter designed for grouping things seems like a fine place to start — mdaniel, Feb 05 '21 at 20:37

score 3 · Accepted Answer · answered Feb 05 '21 at 21:44

The task below creates the list Services including the servers

- set_fact:
    Services: "{{ Services|default([]) + Service|from_yaml }}"
  vars:
    Service: |
      {% for Port in item.1.Port.split(',') %}
      - {{ item.0 }},{{ item.1.Protocol }},{{ Port }},{{ oi.Type }},{{ oi.Env }},{{ oi.GroupName }}
      {% endfor %}
  with_nested:
    - "{{ oi.Server }}"
    - "{{ oi.Service }}"

  Services:
  - 192.168.1.1,TCP,443,Ingress,prod,Test1
  - 192.168.1.1,TCP,22,Ingress,prod,Test1
  - 192.168.1.1,TCP,53,Ingress,prod,Test1
  - 192.168.1.1,UDP,21,Ingress,prod,Test1
  - 192.168.1.2,TCP,443,Ingress,prod,Test1
  - 192.168.1.2,TCP,22,Ingress,prod,Test1
  - 192.168.1.2,TCP,53,Ingress,prod,Test1
  - 192.168.1.2,UDP,21,Ingress,prod,Test1
  - 192.168.1.3,UDP,9996,Egress,dev,Test2
  - 192.168.1.3,UDP,9997,Egress,dev,Test2
  - 192.168.1.4,UDP,9996,Egress,dev,Test2
  - 192.168.1.4,UDP,9997,Egress,dev,Test2

Thank you, its working fine with that, meanwhile i am looking to loop on my principal task with a list a of JSON files, but the problem is that **Services** keept old values everytime on each lookup. i tried to override it in the main task but i get the below error: `fatal: [localhost]: FAILED! => { "msg": "Unexpected templating type error occurred on (Services: "{{ Services|default([]) + Service|from_yaml }}"): coercing to Unicode: need string or buffer, list found" }` Do you have an idea for how to deal with that please ? — Western__TN, Feb 23 '21 at 22:43
[edit] the question and make the case [mre]. Then delete the comment, please. — Vladimir Botka, Feb 24 '21 at 02:37

Ansible : multy loop into json file

1 Answers1