Field [] used in expression does not exist in mappings

Question

The feature I try to fullfit is to create a metric in kibana that display the number of users "unvalidated". I send a log sent when a user registers, then a log when a user is validated.

So the count I want is the difference between the number of registered and the number of validated.

In kibana I cannot do such a math operation, so I found a workaround: I added a "scripted field" named "unvalidated" which is equal to 1 when a user registers and -1 when a user validates his account. The sum of the "unvalidated" field should be the number of unvalidated users.

This is the script I defined in my scripted field: doc['ctxt_code'].value == 1 ? 1 : doc['ctxt_code'].value == 2 ? -1 : 0

with:

ctxt_code 1 as the register log
ctxt_code 2 as the validated log

This setup works well when all my logs have a "ctxt_code", but when a log without this field is pushed kibana throws the following error:

Field [ctxt_code] used in expression does not exist in mappings

I can't understand this error because kibana says:

If a field is sparse (only some documents contain a value), documents missing the field will have a value of 0

which is the case.

Anyone has a clue ?

If you're french, you can help us on the Elasticsearch french forum : https://discuss.elastic.co/t/kibana-probleme-de-scripted-field-qui-nexiste-soit-disant-pas-dans-le-mapping/50618 — lenybernard, May 21 '16 at 14:27
Are you certain that the field `ctxt_code` exists in the mappings of all your logstash indices? I see at the top right of the screenshot that you're working on the last seven days, can you make sure that the logstash indices of the last seven days have that `ctxt_code` field in the created mapping? — Val, May 25 '16 at 03:54

score 2 · Answer 1 · answered Jul 22 '16 at 21:12

It's OK to have logs without the ctxt_code field... but you have to have a mapping for this field in your indices. I see you're querying multiple indices with logstash-*, so you are probably hitting one that does not have it.

You can include a mapping for your field in all indices. Just go into Sense and use this:

PUT logstash-*/_mappings/[your_mapping_name]
{
  "properties": {
    "ctxt_code": {
      "type": "short",           // or any other numeric type, including dates
      "index": "not_analyzed"    // Only works for non-analyzed fields.
    }
  }
}

If you prefer you can do it from the command line: CURL -XPUT 'http://[elastic_server]/logstash-*/_mappings/[your_mapping_name]' -d '{ ... same JSON ... }'

Field [] used in expression does not exist in mappings

1 Answers1