How to run JUnit tests on multiple Ansible assertions

Question

I've got a GitLab pipeline that runs some Ansible playbooks in a container. At the end, I want to run some tests to ensure that everything is properly configured, e.g.

- name: Get the Frobzy value
  raw: "frobzy -x foobar"
  register: frobzy_val
  changed_when: false

- name: Check that Frobzy is valid
  assert:
    that:
      - frobzy_val.stdout is match "oogah"
    fail_msg: "bad frobzy"
  ignore_errors: true # see comments

- name: Get the Blobzy value
  raw: "blobzy -y barfoo"
  register: blobzy_val
  changed_when: false

- name: Check that Blobzy is valid
  assert:
    that:
      - blobzy_val.stdout is match "warra"
    fail_msg: "bad blobzy"
  ignore_errors: true # see comments

This is all nicely configured so that I get a JUnit report that gets displayed on my GitLab pipeline results.

The problem is, let's say both of these tests fail. If I look at the test results from my pipeline, it correctly displays that the tests failed. Yet the pipeline itself is marked as having passed! If I hadn't taken a detailed look at the test results, I would have assumed that everything is tickety-boo, and merged some potentially broken code.

The reason why the pipeline is marked as successful is because of those ignore_errors: true settings. It registers an error, but follows the directive that this should not fail the pipeline. But if I set ignore_errors: false (which is the default behavior), then as soon as the Frobzy test fails, the pipeline will abort and show a failure on the Frobzy test, but since it never ran the Blobzy test, I don't know that Blobzy would have failed, too.

Basically, I want a way to run all my Ansible assertions, even if some of them fail; I want to see all the test results in my pipeline, and I want the pipeline to fail if any tests failed.

How can I do this?

codeaprendiz · Answer 1 · 2022-09-13T15:33:55.103

Here's a quick solution that might help.
Have a task to check if any of the tasks failed. Set a failed variable to true in that case.
Fail the playbook at the end if this variable is set to true.


- hosts: localhost
  vars:
    failed: false
  gather_facts: no
  tasks:

  - name: Get the Frobzy value
    raw: "echo frobzy"
    register: frobzy_val
    changed_when: false

  - name: Check that Frobzy is valid
    assert:
      that:
        - frobzy_val.stdout is match "frobzy"
      fail_msg: "bad frobzy"
    ignore_errors: true # see comments

  - name: Get the Blobzy value
    raw: "echo blobzy"
    register: blobzy_val
    changed_when: false

  - name: Check that Blobzy is valid
    assert:
      that:
        - blobzy_val.stdout is match "blobzy"
      fail_msg: "bad blobzy"
    ignore_errors: true # see comments


  - name: Set a variable to true if the task fails
    set_fact:
      failed: true
    ignore_errors: true
    when: not ("'blobzy' in blobzy_val.stdout" ) or 
          not ("'frobzy' in frobzy_val.msg")


  - name: Fail if failed is true
    fail:
      msg: "Failed"
    when: failed

Run

─ ansible-playbook test.yaml 

PLAY [localhost] ********************************************************************************************************************************************************************************

TASK [Get the Frobzy value] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [Check that Frobzy is valid] ***************************************************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Get the Blobzy value] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [Check that Blobzy is valid] ***************************************************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Set a variable to true if the task fails] *************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail if failed is true] *******************************************************************************************************************************************************************
skipping: [localhost]

PLAY RECAP **************************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

This is a great idea, thanks! Another wrinkle: there are several different playbooks with different tests. Is there a way to share the `failed` variable between playbooks? — Shaul Behr, Sep 14 '22 at 09:40
If you are trying to persist a variable evaluated at runtime between playbooks then this might help :) [stackoverflow-question](https://stackoverflow.com/questions/33896847/how-do-i-set-register-a-variable-to-persist-between-plays-in-ansible). — codeaprendiz, Sep 14 '22 at 11:04
I think I can do this using `vars` in my main playbook (that calls all the tests), and in each playbook I should be able to do a `set_fact` if any tests fail. Is it necessary to do the tests twice, though - once for the `assert` and once for the `set_fact`? It would be really nice if I could set the fact inside the assert (i.e. if it fails), especially because some of my assertions have `with_items` lists. — Shaul Behr, Sep 14 '22 at 11:54
Yeah I was myself wondering the same thing. But `assert` and `set_fact` are different modules in ansible and you cannot set_fact in the same task where you assert. [assert](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/assert_module.html) [set_fact](https://docs.ansible.com/ansible/latest/collections/ansible/builtin/set_fact_module.html). Also it does make sense to keep the functionality that way. In our use case we are trying to squeeze something out of assert :), which is not in it's list of features. It can only tell us if the assertion went good or bad. — codeaprendiz, Sep 14 '22 at 12:23
Any idea how to set a fact matching an assertion with `with_items`? — Shaul Behr, Sep 14 '22 at 13:19
[this link](https://www.reddit.com/r/ansible/comments/c0bfks/ansible_27_is_it_possible_to_pass_list_to_assert/) might help. You can have the assertions in a variable of type list and then loop over them as shown in the example. — codeaprendiz, Sep 14 '22 at 13:25

Zeitounator · Answer 2 · 2022-09-13T23:43:03.600

Make a single assertion for both conditions and delay it until the end so that it fails after the two tests are done.
Add a debug task to show results for your pipelines.

- name: Get the Frobzy value
  raw: "frobzy -x foobar"
  register: frobzy_val
  changed_when: false

- name: Get the Blobzy value
  raw: "blobzy -y barfoo"
  register: blobzy_val
  changed_when: false

- name: Show results for Frobzy and Blobzy
  debug:
    msg:
      - "Frobzy is {{ 'ok' if frobzy_val.stdout is match 'oogah' else 'bad' }}"
      - "Blobzy is {{ 'ok' if lobzy_val.stdout is match 'warra' else 'bad' }}"
  
- name: Fail if any test is bad
  assert:
    that:
      - frobzy_val.stdout is match "oogah"
      - blobzy_val.stdout is match "warra"
    fail_msg: "At least one of Frobzy or Blobzy has bad results. See debug above"
``

How to run JUnit tests on multiple Ansible assertions

2 Answers2