Ansible: "You need to install 'jmespath' prior to running json_query filter", but it is installed

Question

I am using Ansible on top of Python3 and Enterprise Linux 8 (Rocky Linux 8).
When I try to use json_query, I am getting the following error:

fatal: [ansible]: FAILED! => {"msg": "You need to install \"jmespath\" prior to running json_query filter"}

But the Python module appears to be installed already:

# dnf install python3-jmespath
Last metadata expiration check: 1:44:38 ago on Mi 02 Nov 2022 12:54:28 CET.
Package python3-jmespath-0.9.0-11.el8.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!

# pip3 install jmespath
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Requirement already satisfied: jmespath in /usr/lib/python3.6/site-packages

I tested jmespath in an interactive Python interpreter, which is running correctly, without error. This seems to confirm that the Python module is installed, and works correctly.

It seems that the problem only occurs when using Ansible.
Here is the playbook I am using for testing:

---
- name: test json_query
  hosts: ansible
  vars:
    data:
      list1:
        one: 
          name: "hello"
        two: 
          name: "world"
  tasks:
    - name: search variable
      ansible.builtin.debug:
        var: item
      loop: "{{ data | community.general.json_query('list1[*].name') }}"

What am I missing?

Answer 1

I was able to figure it out.

# pip3.8 install jmespath
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3.8 install --user` instead.
Collecting jmespath
Installing collected packages: jmespath
Successfully installed jmespath-1.0.1

Explanation:

This system has two versions of python3 installed:

• python3.6
• python3.8

python3.6 appears to be the default version of Python on this system:

# python3 --version
Python 3.6.8

# which python3.8
/bin/python3.8

It seems that:

• Ansible is using the non-default version python3.8
• The module jmespath needs to be installed for both versions separately.

After running the above pip3.8 command, the error was gone.

Answer 2

Ansible tasks are actions transferred and executed on the remote servers, their ansible_python_interpreter is discovered, and it likely is "platform-python" (python3.6 on RHEL8). You need to make sure the required module is installed there if needed by the module (check ansible-doc ).

However Ansible callback filters, like json_query, run on the Ansible control node. And that may be using another version of Ansible, even if you manage the same machine with ansible.

On the controller you might use a package named ansible or ansible-core, or you might have installed ansible in a python3 venv. In any case jmespath should be installed in the same python version that runs your playbook on the controller.

Recent changes in the ansible-core package complicate this dependency:

Ansible-core 2.13 depends on python 3.9 which installs pip as a weak dependency:

Ansible-core 2.14 depends on python 3.11 which does not install pip, at least on AlmaLinux 8.7.

On AlmaLinux 8.7 there is no python3.11-pip RPM, the same might apply to RHEL8.7 and RockyLinux 8.7.

But in the 8.8 release pip re-appears!

If your build depends on AlmaLinux8 your (providers') package repositories might have, or might not have the RPMs needed to install pip. And whether you can access Pypi depends on your situation as well.

Note that pip3 points to platform-python on RHEL8, which is Python3.6, that is fine for python libraries required by modules, but not for jmespath. You better be more explicit!

If you run ansible-core 2.14 with python3.11 then you can install jmespath with:

/usr/bin/pip3.11 install jmespath

If you run ansible-core 2.12 with python3.9 then you can install `jmespath with:

/usr/bin/pip3.9 install jmespath

Conclusion Batteries seem too heavy to include nowadays, the json_query plugin is not a part of ansible-core, but included in the collection community.general, it depends on an external library jmespath that you need to install on the controller with python3 -m pip install jmespath. You might not be successful for various reasons. There might be other ways to post-process the JSON return values on the controller, like the jq program.

https://github.com/ansible/ansible/blob/56f2e65ee78bd0031775e986ff45501b590d9dc1/docs/docsite/rst/playbook_guide/playbooks_filters.rst#selecting-json-data-json-queries

Answer 3

The Ansible tasks are executed on the remote server. You need to make sure the required module is installed there.

Answer 4

I confused my virtualenv with the laptops's python/pip binary I installed this module for every python version available on my laptop ( python3.8 / 3.9 / 3.10 )

Then I noticed after a while, that I'm using Ansible from within a virtualenv directory, and therefor - i needed to install from that local pip: venv/bin/pip3 install jmespath ...