When to use from_json filter in Ansible?

Question

When should I use the from_json filter in Ansible?

I found out that using it sometimes has and sometimes have no effect.

Please consider the following example which illustrates the inconsistency I am getting.

^{Included in reverse order are: the questions - expected result - actual result - the playbook - the data. The data is taken from this question and the playbook is based on this answer.}

• The question(s):

  Why storing the left part (before json_query) of the following expression in a variable and then using json_query on the variable causes the expression to be evaluated differently?

  "{{ lookup('file','test.json') | json_query(query) }}"
  

  Why does adding from_json filter alter the results (but does not if processing a variable):

  "{{ lookup('file','test.json') | from_json | json_query(query) }}"
  

• Expected result:

  Last four tasks should give the same result. Alternatively, last two tasks should give the same result as previous two tasks.

• Actual result (last four tasks only):

  One task result differs.

  TASK [This query is run against lookup value with from_json stored in a variable] ***
  ok: [localhost] => {
      "msg": [
          678
      ]
  }
  
  TASK [This query is run against lookup value without from_json stored in a variable] ***
  ok: [localhost] => {
      "msg": [
          678
      ]
  }
  
  TASK [This query is run directly against lookup value with from_json] **********
  ok: [localhost] => {
      "msg": [
          678
      ]
  }
  
  TASK [This query is run directly against lookup value without from_json - the result is empty - why?] ***
  ok: [localhost] => {
      "msg": ""
  }
  

• The playbook:

  ---
  - hosts: localhost
    gather_facts: no
    connection: local
    tasks:   
      - set_fact:
          from_lookup_with_from_json: "{{ lookup('file','test.json') | from_json }}"
  
      - set_fact:
          from_lookup_without_from_json: "{{ lookup('file','test.json') }}"
  
      - name: Save the lookup value stored in a variable in a file for comparison
        copy: content="{{ from_lookup_with_from_json }}" dest=./from_lookup_with_from_json.txt
  
      - name: Save the lookup value stored in a variable in a file for comparison (they are the same)
        copy: content="{{ from_lookup_without_from_json }}" dest=./from_lookup_without_from_json.txt
  
      - name: This query is run against lookup value with from_json stored in a variable
        debug: msg="{{ from_lookup_with_from_json | json_query(query) }}"
        vars:
          query: "Foods[].{id: Id, for: (Tags[?Key=='For'].Value)[0]} | [?for=='Tigger'].id"
  
      - name: This query is run against lookup value without from_json stored in a variable
        debug: msg="{{ from_lookup_without_from_json | json_query(query) }}"
        vars:
          query: "Foods[].{id: Id, for: (Tags[?Key=='For'].Value)[0]} | [?for=='Tigger'].id"
  
      - name: This query is run directly against lookup value with from_json
        debug: msg="{{ lookup('file','test.json') | from_json | json_query(query) }}"
        vars:
          query: "Foods[].{id: Id, for: (Tags[?Key=='For'].Value)[0]} | [?for=='Tigger'].id"
  
      - name: This query is run directly against lookup value without from_json - the result is empty - why?
        debug: msg="{{ lookup('file','test.json') | json_query(query) }}"
        vars:
          query: "Foods[].{id: Id, for: (Tags[?Key=='For'].Value)[0]} | [?for=='Tigger'].id"
  

• The data (test.json):

  { "Foods" :
    [ { "Id": 456
      , "Tags":
        [ {"Key":"For", "Value":"Heffalump"}
        , {"Key":"Purpose", "Value":"Food"}
        ]
      }
    , { "Id": 678
      , "Tags":
        [ {"Key":"For", "Value":"Tigger"}
        , {"Key":"Purpose", "Value":"Food"}
        ]
      }
    , { "Id": 911
      , "Tags":
        [ {"Key":"For", "Value":"Roo"}
        , {"Key":"Purpose", "Value":"Food"}
        ]
      }
    ]
  }
  

Answer 1

json_query requires Python object (dict) as input, if you feed it with string, it gives empty string as result.

You get different result because of Ansible templating engine tricky work.
I should definitely write a post about it on my site...

After evaluating jijna2 expression Ansible try to cast complex types to Python objects (like dict or list). See my other answer.

In your case:

1.

- set_fact:
    from_lookup_with_from_json: "{{ lookup('file','test.json') | from_json }}"

from_lookup_with_from_json is a dict, because you manually convert JSON-string from file to dict with from_json filter.

2.

- set_fact:
    from_lookup_without_from_json: "{{ lookup('file','test.json') }}"

from_lookup_with_from_json becomes dict, because Ansible converts it when jinja2 expression ends with }}. So from_json is actually unnecessary as the last filter in chain.

3.

  debug: msg="{{ lookup('file','test.json') | from_json | json_query(query) }}"

Again, you manually convert JSON-string here. So json_query get dict as input.

4.

  debug: msg="{{ lookup('file','test.json') | json_query(query) }}"

In this case you feed JSON-string (not dict) as input to json_query filter. As everything happens inside one jinja2 expression, Ansible doesn't attempt to convert anything in between.

You can also get empty string result with a variable this way:

- set_fact:
    from_lookup_force_string: "{{ lookup('file','test.json') | string }}"

In this case from_lookup_force_string will not be converted by Ansible tempating engine, and json_query will give you empty response on it.