Ansible is Unable to read filenames having white spaces passed by Jenkins multiline parameter

Question

Ansible code works fine & is able to read line-by-line when the passed filenames has no white spaces.

The issue is when Jenkins passes the filenames in source_file parameter having whitespaces as below.

Below is how ansible is called:

sh "ansible-playbook -i /web/playbooks/filecopy/allmwhosts.hosts /web/playbooks/filecopy/copyfiles.yml -e source_host=$source_host -e '{ source_file: $source_file }'

[Pipeline] sh (hide)
+ ansible-playbook -i /web/filecopy/allmwhosts.hosts /web/playbooks/filecopy/copyfiles.yml -e source_host=usurb31as30v -e '{ source_file: /web/lib/hello.txt
/web/lib/Policy Form_LifeAgents_new.pdf
/web/lib/Policy Form_LifeAgents_old.pdf }' 

Ansible code below:

   - debug:
       msg: "FILES: {{ files_list }}"

   - debug:
       msg: "The NEWLINE files are {{ item }}"
     with_items:
       - "{{ source_file.split() }}"

   - debug:
       msg: "The NEWLINE1 files are {{ item }}"
     with_items:
       - "{{ source_file.splitlines() }}"

   - set_fact:
       source_file_new: "{{ item | quote + '\n' + source_file | default() }}"
     with_items:
       - "{{ source_file.split('\n') }}"

   - debug:
       msg: "AFTER QUOTE: {{ source_file_new }}"


   - debug:
       msg: "QUOTED NEWLINE files are {{ item }}"
     with_items:
       - "{{ source_file_new.split('\n') }}"

Output:

TASK [debug] *******************************************************************
task path: /web/playbooks/filecopy/copyfiles.yml:18
Saturday 20 August 2022  22:59:50 -0500 (0:00:00.077)       0:00:00.118 ******* 
ok: [localhost] => {
    "msg": "FILES: /web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf"
}

TASK [debug] *******************************************************************
task path: /web/playbooks/filecopy/copyfiles.yml:21
Saturday 20 August 2022  22:59:50 -0500 (0:00:00.032)       0:00:00.150 ******* 
ok: [localhost] => (item=/web/lib/hello.txt) => {
    "msg": "The NEWLINE files are /web/lib/hello.txt"
}
ok: [localhost] => (item=/web/lib/Policy) => {
    "msg": "The NEWLINE files are /web/lib/Policy"
}
ok: [localhost] => (item=Form_LifeAgents_new.pdf) => {
    "msg": "The NEWLINE files are Form_LifeAgents_new.pdf"
}
ok: [localhost] => (item=/web/lib/Policy) => {
    "msg": "The NEWLINE files are /web/lib/Policy"
}
ok: [localhost] => (item=Form_LifeAgents_old.pdf) => {
    "msg": "The NEWLINE files are Form_LifeAgents_old.pdf"
}
    
TASK [debug] *******************************************************************
task path: /web/playbooks/filecopy/copyfiles.yml:21
Sunday 21 August 2022  12:16:24 -0500 (0:00:00.021)       0:00:00.101 ********* 
ok: [localhost] => (item=/web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf /web/lib/Policy Specimen_MLA_Everest_new.pdf /web/lib/Policy Specimen_MLA_Everest_old.pdf) => {
    "msg": "The NEWLINE1 files are /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf /web/lib/Policy Specimen_MLA_Everest_new.pdf /web/lib/Policy Specimen_MLA_Everest_old.pdf"
}

TASK [set_fact] ****************************************************************
task path: /web/playbooks/filecopy/copyfiles.yml:26
Saturday 20 August 2022  22:59:50 -0500 (0:00:00.069)       0:00:00.219 ******* 
ok: [localhost] => (item=/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf) => {
    "ansible_facts": {
        "source_file_new": "'/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf'\n/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf"
    },
    "ansible_loop_var": "item",
    "changed": false,
    "item": "/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf"
}

TASK [debug] *******************************************************************
task path: /web/playbooks/filecopy/copyfiles.yml:31
Saturday 20 August 2022  22:59:50 -0500 (0:00:00.080)       0:00:00.300 ******* 
ok: [localhost] => {
    "msg": "AFTER QUOTE: '/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf'\n/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf"
}

TASK [debug] *******************************************************************
task path: /web/playbooks/filecopy/copyfiles.yml:35
Saturday 20 August 2022  22:59:50 -0500 (0:00:00.031)       0:00:00.331 ******* 
ok: [localhost] => (item='/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf') => {
    "msg": "QUOTED NEWLINE files are '/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf'"
}
ok: [localhost] => (item=/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf) => {
    "msg": "QUOTED NEWLINE files are /web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf"
}

The expectation was three lines should have been read by ansible however neither the split() or split('\n') is helping

Answer 1

You are launching your Ansible playbook through shell. The problem here is a quoting issue. (ba)sh interprets new lines and spaces as parameter separators. So you need to quote everything correctly in order for:

1. (ba)sh to understand the -e parameter to Ansible ends after the last file in the list
2. Ansible to understand that this is a single string containing new lines and spaces.

Once this is fixed the rest of the processing is trivial since you just have to source_file.splitlines() your value to get your list of individual files.

Note that Jenkins has an Ansible plugin that can ease your life in such cases on the gui. If you add a "Invoke ansible playbook" build step and click on "advanced", you will be able to add extra variables by simply entering the key name and the value (which can be a variable). You can see the correct secure quoting in the result when launching your job.

Unfortunately, this handy help is not available in the pipeline syntax helper where you will also have to construct your string of extra parameters by yourself.

Long story short: this is how your shell script command should be written to securely quote all your ansible parameters:

ansible-playbook -i /web/playbooks/filecopy/allmwhosts.hosts \
/web/playbooks/filecopy/copyfiles.yml \
-e "source_host='$source_host'" \
-e "source_file='$source_file'"

As a proof of concept, here is an all-in-one demo declarative Jenkins pipeline that you can paste in a pipeline job and which successfully runs the in-declared playbook both through shell and the Ansible plugin.

Note that since I declared the job parameters inside the pipeline, the first build will systematically fail. You will only be able to provide the params interactively on the second build. The defaults should be enough to reproduce the situation described in your question.

pipeline {
    agent any
    
    stages {
        stage('Setup parameters') {
            steps {
                script {
                    properties([
                        parameters([
                            text(
                                defaultValue: "/a/file\n/an/other file\n/one/more/file", 
                                description: "Enter absolute path of files on target one on each line", 
                                name: "source_file"
                            )
                        ])
                    ])
                }
            }
        }
        
        stage('Create playbook') {
            steps {
                sh '''
cat <<"EOF" > playbook.yml
---
- hosts: localhost
  gather_facts: false
                  
  vars:
    file_list: "{{ source_file.splitlines() }}"
                  
  tasks:
    - name: Show content of processed incoming extra var
      debug:
        var: file_list
        
    - name: Just proove it works
      debug:
        msg: "received individual file {{ item }}"
      loop: "{{ file_list }}"
EOF
'''
            }
        }
        
        stage('Run playbook through shell'){
            steps {
                sh '''
                ansible-playbook playbook.yml -e "source_file='$source_file'"
                '''
            }
        }
        
        stage('Run playbook through ansible plugin'){
            steps {
                ansiblePlaybook extras: '-e "source_file=\'$source_file\'"', playbook: 'playbook.yml'
            }
        }
    }
}

Which gives (on second run and using the default values):

Started by user Olivier Clavel
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/testpipe
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Setup parameters)
[Pipeline] script
[Pipeline] {
[Pipeline] properties
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Create playbook)
[Pipeline] sh
+ cat
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Run playbook through shell)
[Pipeline] sh
+ ansible-playbook playbook.yml -e source_file='/a/file
/an/other file
/one/more/file'
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [localhost] ***************************************************************

TASK [Show content of processed incoming extra var] ****************************
ok: [localhost] => {
    "file_list": [
        "/a/file",
        "/an/other file",
        "/one/more/file"
    ]
}

TASK [Just proove it works] ****************************************************
ok: [localhost] => (item=/a/file) => {
    "msg": "received individual file /a/file"
}
ok: [localhost] => (item=/an/other file) => {
    "msg": "received individual file /an/other file"
}
ok: [localhost] => (item=/one/more/file) => {
    "msg": "received individual file /one/more/file"
}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Run playbook through ansible plugin)
[Pipeline] ansiblePlaybook
[testpipe] $ ansible-playbook playbook.yml -e "source_file='/a/file
/an/other file
/one/more/file'"
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [localhost] ***************************************************************

TASK [Show content of processed incoming extra var] ****************************
ok: [localhost] => {
    "file_list": [
        "/a/file",
        "/an/other file",
        "/one/more/file"
    ]
}

TASK [Just proove it works] ****************************************************
ok: [localhost] => (item=/a/file) => {
    "msg": "received individual file /a/file"
}
ok: [localhost] => (item=/an/other file) => {
    "msg": "received individual file /an/other file"
}
ok: [localhost] => (item=/one/more/file) => {
    "msg": "received individual file /one/more/file"
}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Answer 2

You have the individual files in multiple lines. Your command may look like passing multiple lines to your Ansible playbook, but I think the variable source_file actually arrives as a string with no newlines (\n), so the paths are just separated by a space.

Assuming you have spaces in paths and all paths are absolute (start with /), I would do the splitting as follows:

- name: Split source_file
  set_fact:
    source_file_list: '{{ source_file | regex_findall("(/(?:[\w/\.-]+(?: [^/])?)+)", "\1") }}'

The regex matches the starting / followed by the allowed characters (\w, /, ., -). The space character is allowed only if it is followed by a character that is not a /.

Here is my code with single-line input (separated by spaces) and corresponding output.

- name: List source_file raw input
  debug: var=source_file

- name: Split source_file
  set_fact:
    source_file_list: '{{ source_file | regex_findall("(/(?:[\w/\.-]+(?: [^/])?)+)", "\1") }}'

- name: Output source_file_list with splitted result
  debug: var=source_file_list

TASK [List source_file raw input] **************************************************************************************
ok: [localhost] => {
    "source_file": "/web/lib/hello.txt /web/lib/Policy Form_LifeAgents_new.pdf /web/lib/Policy Form_LifeAgents_old.pdf"
}

TASK [Split source_file] ***********************************************************************************************
ok: [localhost]

TASK [Output source_file_list with splitted result] ********************************************************************
ok: [localhost] => {
    "source_file_list": [
        "/web/lib/hello.txt",
        "/web/lib/Policy Form_LifeAgents_new.pdf",
        "/web/lib/Policy Form_LifeAgents_old.pdf"
    ]
}