elastic search ngram special characters

Question

I am having a elastic search node with the following default config

index :
  analysis :
    analyzer :
      default_index :
        type : custom
        tokenizer : whitespace
        filter :
        - lowercase
        - asciifolding
        - stop
        - my_ngram
        char_filter : html_strip
      default_search:
        type : custom
        tokenizer :  whitespace
        filter:
        - lowercase
        - asciifolding
        - stop
        char_filter :  html_strip
    filter:
      my_ngram:
        type: nGram
        max_gram: 50

I then create a index "test"

curl -XPUT localhost:9200/test -d '{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  }
}'

I posted

curl -XPOST localhost:9200/test/sub -d '{"n1" : "so?:me"}'

search as

curl -XPOST 'localhost:9200/test/sub/_search?pretty&q=\?'

and I get the right result with the above entry shown, but when I do

curl -XPOST localhost:9200/test/sub/_search -d '{
  "query": {
    "query_string": {
      "query": "\?"
    }
  }
}'

I get an exception as below

{
  "error": "SearchPhaseExecutionException[Failed to execute phase [query_fetch], total failure;
            shardFailures {[1fLLfu79Qou8RbdrI6y8qw][test][0]: 
            SearchParseException[[test][0]: from[-1],size[-1]: 
            Parse Failure [Failed to parse source [
              {
                "query": {
                  "query_string": {
                    "query": "\\?"
                  }
                }
              }
            ]]];
            nested: QueryParsingException[[test] Failed to parse]; 
            nested: JsonParseException[Unrecognized character escape '?' (code 63)\n at [Source: [B@1601cda; line: 1, column: 45]]; }]",
  "status": 500
}

I am not sure what I am missing here?

some more detail, I found and it's more confusing.

if I post

curl -XPOST localhost:9200/test/sub/_search -d '{
  "query": {
    "query_string": {
      "query": "\\?"
    }
  }
}'

I get the result back, properly, looks like the JSON escape character has to be escaped itself. but then I post

curl -XPOST localhost:9200/test/sub -d '{"n1" : "oi\\me"}'

and now if I post

curl -XPOST localhost:9200/test/sub/_search?pretty -d '{
  "query": {
    "query_string": {
      "query": "\\\\"
    }
  }
}'

I get the result, assuming what I found previously the above represents just the first '\' in the answer it shows so ideally

curl -XPOST localhost:9200/test/sub/_search?pretty -d '{
  "query": {
    "query_string": {
      "query": "\\\\\\\\"
    }
  }
}'

should work but it doesn't. so very confused.

flamecto · Answer 1 · 2013-08-01T19:48:23.813

I think it's because in oi\\me, the first backward slash is used to escape the second one rather than stored as a literal character. This explains why \\\\ works, because in the HTTP request, two of the slashes escape the other two, and then in the query, first of the remaining ones escapes the second.

As a general rule, you have to escape more when you pass the query as JSON. That is,

curl -XPOST 'localhost:9200/test/sub/_search?pretty&q=\?'

is the same as

curl -XPOST 'localhost:9200/test/sub/_search?pretty' -d '{"query" : {"query_string" : {"query" : "\\?"}}}'

score 0 · Answer 2 · answered Aug 02 '13 at 10:44

Yeh your representation of the escape is correct and i got that working yesterday late but was still not able to get "\" search correctly, for json search post we need an additional "\" hence for the first selection i will post

curl -XPOST 'localhost:9200/test/sub/_search?pretty' -d '{"query" : {"query_string" : {"query" : "\\"}}}'

but that doesnot help have to use "\\" and cannnot use more or less, so trying to figure out whats the query for matching single "\" and "\", though with elastic utils in python if i just escape every special character with "\" it works great via code for all special characters includeing "\" but curl doesnot

elastic search ngram special characters

2 Answers2