10

I am reading a base64 file from HashiCorp’s vault with the help of the hashi_vault module. Sample of code:

- name: Vault get b64.pfx file
  set_fact:
      b64_pfx: "{{ lookup('hashi_vault',
                    'secret={{ path_pfx }} token={{ token }} url={{ url }} cacert={{ role_path}}/files/CA.pem')}}"

Then as a next step I need to decode this base64 var to a binary format and store it in in a file. I am currently using shell module to do the work. Sample of code:

- name: Decode Base64 file to binary
  shell: "echo {{ b64_pfx }} | base64 --decode > {{ pfxFile }}"
  delegate_to: localhost

I was looking online for possible solutions e.g. ( Copy module with base64-encoded binary file adds extra character and How to upload encrypted file using ansible vault?).

But the only working solution that I can found is using the shell module. Since this is an old problem is there any workaround on this?

Update:

Do not use Python 2.7 as there seems to be a bug on the b64decode filter (sample below):

<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1573819503.84-50241917358990 `" && echo ansible-tmp-1573819503.84-50241917358990="` echo /tmp/ansible-tmp-1573819503.84-50241917358990 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/commands/command.py
<localhost> PUT /tmp/ansible-local-18pweKi1/tmpjQGOz8 TO /tmp/ansible-tmp-1573819503.84-50241917358990/AnsiballZ_command.py
<localhost> EXEC /bin/sh -c 'chmod u+x /tmp/ansible-tmp-1573819503.84-50241917358990/ /tmp/ansible-tmp-1573819503.84-50241917358990/AnsiballZ_command.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /tmp/ansible-tmp-1573819503.84-50241917358990/AnsiballZ_command.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /tmp/ansible-tmp-1573819503.84-50241917358990/ > /dev/null 2>&1 && sleep 0'
changed: [hostname -> localhost] => {
    "changed": true,
    "cmd": "shasum -a 1 /tmp/binary_file\nshasum -a 1 /tmp/binary_file.ansible\n",
    "delta": "0:00:00.126279",
    "end": "2019-11-15 13:05:04.227933",
    "invocation": {
        "module_args": {
            "_raw_params": "shasum -a 1 /tmp/binary_file\nshasum -a 1 /tmp/binary_file.ansible\n",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": true
        }
    },
    "rc": 0,
    "start": "2019-11-15 13:05:04.101654",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "4a71465d449a0337329e76106569e39d6aaa5ef0  /tmp/binary_file\nead5cb632f3ee80ce129ef5fe02396686c2761e0  /tmp/binary_file.ansible",
    "stdout_lines": [
        "4a71465d449a0337329e76106569e39d6aaa5ef0  /tmp/binary_file",
        "ead5cb632f3ee80ce129ef5fe02396686c2761e0  /tmp/binary_file.ansible"
    ]
}

Solution: use Python 3 with b64decode filter (sample below):

<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp/ansible-tmp-1573819490.9511943-224511378311227 `" && echo ansible-tmp-1573819490.9511943-224511378311227="` echo /tmp/ansible-tmp-1573819490.9511943-224511378311227 `" ) && sleep 0'
Using module file /usr/local/lib/python3.6/site-packages/ansible/modules/commands/command.py
<localhost> PUT /tmp/ansible-local-18epk_0jsv/tmp4t3gnm7u TO /tmp/ansible-tmp-1573819490.9511943-224511378311227/AnsiballZ_command.py
<localhost> EXEC /bin/sh -c 'chmod u+x /tmp/ansible-tmp-1573819490.9511943-224511378311227/ /tmp/ansible-tmp-1573819490.9511943-224511378311227/AnsiballZ_command.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /tmp/ansible-tmp-1573819490.9511943-224511378311227/AnsiballZ_command.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /tmp/ansible-tmp-1573819490.9511943-224511378311227/ > /dev/null 2>&1 && sleep 0'
changed: [hostname -> localhost] => {
    "changed": true,
    "cmd": "shasum -a 1 /tmp/binary_file\nshasum -a 1 /tmp/binary_file.ansible\n",
    "delta": "0:00:00.135427",
    "end": "2019-11-15 13:04:51.239969",
    "invocation": {
        "module_args": {
            "_raw_params": "shasum -a 1 /tmp/binary_file\nshasum -a 1 /tmp/binary_file.ansible\n",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": true
        }
    },
    "rc": 0,
    "start": "2019-11-15 13:04:51.104542",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "4a71465d449a0337329e76106569e39d6aaa5ef0  /tmp/binary_file\n4a71465d449a0337329e76106569e39d6aaa5ef0  /tmp/binary_file.ansible",
    "stdout_lines": [
        "4a71465d449a0337329e76106569e39d6aaa5ef0  /tmp/binary_file",
        "4a71465d449a0337329e76106569e39d6aaa5ef0  /tmp/binary_file.ansible"
    ]
}

Since Python 2 is reaching the end of life in (January 1, 2020) there is no point of raising the bug.

Thanos
  • 1,618
  • 4
  • 28
  • 49
  • 2
    have you come across this? [https://terryhowe.github.io/ansible-modules-hashivault/modules/hashivault_read_to_file_module.html](https://terryhowe.github.io/ansible-modules-hashivault/modules/hashivault_read_to_file_module.html). synopsis: Reads and deocdes a base64 encoded file from Hashicorp Vault and saves it to disk. – ilias-sp Nov 13 '19 at 20:31
  • @ilias-sp I did read it initially I could give it a try. Thanks for reminding :) – Thanos Nov 14 '19 at 13:52

2 Answers2

15

Using the b64decode filter at least on ansible 2.9 does what you want:

- copy:
    dest: '{{ pfxFile }}'
    content: '{{ b64_pfx | b64decode }}'
  delegate_to: localhost

I confirmed it writes only the specified bytes (no trailing whitespace) and is binary safe.

If you tried that behavior, and it doesn't work for you, then update your question to say that and to include the version of ansible you are using. I also think that bug you linked to has been fixed, because I tried their exact case on ansible 2.9 and it did the right thing:

- hosts: localhost
  connection: local
  gather_facts: no
  tasks:
  - set_fact:
      string_in_base64: 'sxZARwIVokeqOMGPygc1S20CaGPiKDRGRzg0oSVGmCF2oXHua+9fVhriUQRd8vkmvpHoBmSsI6Y='
  - copy:
      dest: binary_file.ansible
      content: '{{ string_in_base64 | b64decode }}'
  - shell: |
      echo '{{ string_in_base64 }}' | base64 --decode > binary_file
      shasum -a 1 binary_file
      shasum -a 1 binary_file.ansible

{
  "changed": true,
  "cmd": "echo 'sxZARwIVokeqOMGPygc1S20CaGPiKDRGRzg0oSVGmCF2oXHua+9fVhriUQRd8vkmvpHoBmSsI6Y=' | base64 --decode > binary_file\nshasum -a 1 binary_file\nshasum -a 1 binary_file.ansible\n",
  "delta": "0:00:00.162251",
  "end": "2019-11-13 13:10:56.683186",
  "rc": 0,
  "start": "2019-11-13 13:10:56.520935",
  "stderr": "",
  "stderr_lines": [],
  "stdout": "7e88df04cf47019ae22e9c658b62c26b706c6ea5  binary_file\n7e88df04cf47019ae22e9c658b62c26b706c6ea5  binary_file.ansible",
  "stdout_lines": [
    "7e88df04cf47019ae22e9c658b62c26b706c6ea5  binary_file",
    "7e88df04cf47019ae22e9c658b62c26b706c6ea5  binary_file.ansible"
  ]
}
mdaniel
  • 31,240
  • 5
  • 55
  • 58
  • 1
    I just tested your sample of code. On a node_1 I am running ansible 2.8.5 with Python 2.7.5. The binaries differ. Then I tested the code on node_2 with ansible 2.8.6 and Python 3.6.8 and the binaries are matching. I am afraid that I can not update the Ansible version on node_1 but at least I know why it was failing. Thank you for the sample of code I did not thought that the version 2.8.5 might fail. – Thanos Nov 14 '19 at 11:04
  • 1
    *Update*: I launch a container with Python 2.7.5 and Ansible 2.9.0 and the binaries do not match again. Then I installed on the container Python 3.6.8 with Ansible 2.9.1 and it is working as expected. @mdaniel what Python version are you testing? – Thanos Nov 14 '19 at 13:03
  • 1
    Yes, since [python2 is dead](https://www.python.org/dev/peps/pep-0373/#update) I don't use it anywhere, but strangely enough while my **ansible** is hosted in py3.7 it actually whines citing `Platform darwin on host localhost is using the discovered Python interpreter at /usr/bin/python` which is py2.7 (and I confirmed with `-vvvv` that it is in fact exec-ing `/usr/bin/python` for the `copy:` task) – mdaniel Nov 15 '19 at 04:40
  • Yes you are right. Although that I was using the python_interpeter it still was loading first python 2 ansible package and then the ansible was running with Python 3. In result I had the same problem. So I decided to build my container purely with Python 3 to avoid this problem and then it works just fine. – Thanos Nov 15 '19 at 12:22
  • A caveat is that while it works with `content:`, it doesn't work with the `stdin` argument of a `command`, unfortunately (try e.g. `stdin: '{{ "3OY=" | b64decode }}'`, which will get you `UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1` – Clément Nov 13 '22 at 07:03
0

This is simpler than the other answers. Just pipe the function at the end of the lookup()

- name: Vault get b64.pfx file
  set_fact:
      b64_pfx: "{{ lookup('hashi_vault',
                    'secret={{ path_pfx }} token={{ token }} url={{ url }} cacert={{ role_path}}/files/CA.pem') | b64decode }}"
mrk
  • 640
  • 8
  • 16