validate all the files before copying to all the remote servers

Question

I have an ansible playbook which is made up of two plays as shown below:

• First play validates everything on the localbox where I am running my ansible before invoking Play2. If my validation fails in Play1 then I don't want to start Play2 at all.
• Second play will only start if Play1 validation is successful and this play creates a directory on remoter server if it is not there and then copies process.tar.gz file in a particular directory on all remote servers.

Below is my playbook:

---
- name: Play 1
  hosts: 127.0.0.1
  tasks:
      - name: check whether we have all the necessary files
        shell: "ls files/ | wc -l"
        retries: 10
        delay: 10
        register: number_of_files
        until: "number_of_files.stdout == '10'"

      - name: Total number of files
        debug: msg="Total files {{ number_of_files.stdout }}"

      - name: check whether all the files are generated within a minute difference
        shell: "find files/ -mmin -1 -type f -print | wc -l"
        register: file_time
        failed_when: "file_time.stdout != '10'"

      - name: Total number of files
        debug: msg="Total files {{ file_time }}"

      - name: check whether success file was generated
        shell: "ls files/SUCCESS | wc -l"
        register: success_file
        failed_when: "success_file.stdout != '1'"

      - name: Success file
        debug: msg="{{ success_file }}"

      - name: compress all the files in tar.gz
        shell: "rm process.tar.gz; tar -czvf process.tar.gz -C files . --exclude='SUCCESS'"

- name: Play 2
  hosts: ALL_HOSTS
  serial: "{{ serial }}"
  tasks:
      - name: creates directory
        file: path=/data/taks/files/ state=directory

      - name: copy and untar process.tar.gz file
        unarchive: src=process.tar.gz dest=/data/taks/files/

      - name: sleep for few seconds
        pause: seconds=20

I wanted to see if there is any better way to write my above ansible? Since I recently started working with ansible so not sure I am following all the best practices.. I am running ansible version ansible 2.4.3.0. Also I always get a warning from ansible on this task:

- name: compress all the files in tar.gz
  shell: "rm process.tar.gz; tar -czvf process.tar.gz -C files . --exclude='SUCCESS'"

Here is the warning:

[WARNING]: Consider using file module with state=absent rather than running rm
[WARNING]: Consider using unarchive module rather than running tar

Update:

Below is my path where my ansible playbook is sitting and from this directory only, I execute my PLAYBOOK1. And files directory contains all the files including SUCCESS which I need to compress into tar.gz.

jenkins@machineA:~/jobs/processdata/workspace$ ls -lrth
total 145M
drwxr-xr-x 2 jenkins jenkins 4.0K Feb 19 17:36 files
-rw-r--r-- 1 jenkins root    1.6K Feb 19 19:32 PLAYBOOK1.yml
-rw-r--r-- 1 jenkins root    1.6K Feb 19 19:32 PLAYBOOK2.yml

Here is the error I am getting after I execute PLAYBOOK1 with your changes:

TASK [Check all files] **********************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": false, "examined": 0, "failed_when_result": true, "files": [], "matched": 0, "msg": "/files was skipped as it does not seem to be a valid directory or it cannot be accessed\n"}

Answer 1

See my coments marked with #

Playbook1

- hosts: YOURHOST1
  remote_user: "YOURUSER"
  become: True

  tasks:

  #Instead using shell, use find module... It will check if there are 10 files created before 1 minute in folder, otherwise playbook fails
  - name: Check all files
    find: paths=/var/lib/jenkins/jobs/processdata/workspace/files
          file_type=file
          age=-1m
          age_stamp=mtime
    register: files
    failed_when: files.matched < 10

  - name: Number of files
    debug: msg="Total files {{ files.matched }}"

  #Check for SUCCESS file, I didn't add an action to delete SUCCESS file... Hope you have any other script to do it
  - name: Check Success File
    find: paths=/var/lib/jenkins/jobs/processdata/workspace/files
          file_type=file
          patterns=SUCCESS
    register: success_file
    failed_when: success_file.matched < 1

  - name: Success file
    debug: msg='Success file is {{ success_file.files.0.path }}'

  - name: Remove previous tarFile
    file: path=/tmp/process.tar.gz
          state=absent

  #I've found a way to make it work
  - name: compress all the files in tar.gz
    archive: path="{{ files_to_archive }}"
             dest=/tmp/test.tar.gz
             format=gz
    vars:
        files_to_archive: "{{ files.files | map(attribute='path') | list }}"

- include: PLAYBOOK2.yml 

Playbook2 If one of the previous actions fails, Ansible stops and won't run any other action.

- hosts: YOURHOST2
  remote_user: "YOURUSER"
  become: True

  tasks:

  - name: creates directory
    file: path=/data/taks/files/
          state=directory

  - name: Copy tar file
    copy: src=/tmp/test.tar.gz
          dest=/tmp/process.tar.gz
          owner=root
          group=root
          mode=0744

  - name: copy and untar process.tar.gz file
    unarchive: src=/tmp/process.tar.gz
               dest=/data/taks/files/

  - name: Remove SUCCESS File
    file: path=/data/taks/files/SUCCESS
          state=absent

#I don't know why you'll need this:
  - name: sleep for few seconds
    pause: seconds=20

Please consider that one of the features of ansible is idempotency, you should write your scripts to run and continue from previous actions (if they have failed in a previous launch). Neither yours nor mine scripts are idempotent, because you need to find files newer than a minute before, hence during next launch (if something fails) your script discards files older than a minute.

Same happens with SUCCESS file, if you have a platform/script that is continue deleting it, is possible that Ansible script fails when running.