Filter on boolean field increases latency and CPU

Question

I'm using the AWS ElasticSearch service (1.5.2) and is receiving more than 100 requests/sec without problems.

Almost every query have geo filters, full text filters, integer filters... But I have a problem, adding one simple filter more; a boolean term filter.

{
  "from" : 0,
  "size" : 10,
  "query" : {
    "filtered" : {
      "query" : {
        "match_all" : { }
      },
      "filter" : {
        "and" : {
          "filters" : [ {
            "bool" : {
              "must" : [ {
                "terms" : {
                  "my_boolean_field" : [ false ]
                }
              }, 
              {
                "range" : {
                  "_timestamp" : {
                    "from" : null,
                    "to" : "2016-05-04T15:12:00Z",
                    "include_lower" : true,
                    "include_upper" : false
                  }
                }
              } ]
            }
          }, {
            "geo_distance" : {
              "rounded_location" : [ -8.42, 42.24 ],
              "distance" : "300000m",
              "distance_type" : "plane",
              "optimize_bbox" : "indexed"
            }
          } ]
        }
      }
    }
  },
  "sort" : [ {
    "_geo_distance" : {
      "rounded_location" : [ {
        "lat" : 42.24,
        "lon" : -8.42
      } ],
      "unit" : "m"
    }
  }, {
    "date" : {
      "order" : "desc"
    }
  }, {
    "price" : {
      "order" : "asc"
    }
  } ]
}

Once the boolean filter is added, CPU in all nodes of ElasticSearch increases from 10 to 30% and during 10-15 minutes the average latency increases from 20 to 400ms, after a while latencies go back to normal numbers but CPU doesn't.

That boolean field is mapped properly and is searchable... has anyone a hint about this issue? (Query is formed with "terms" instead of "term" due to the official Java library that I'm using).

Can you try the same query, but swap positions between the `terms` boolean filter and the `range` one in the `must` clause? — Andrei Stefan, May 04 '16 at 21:39
Thanks for answering! I tried that, and also I tried to put the boolean out of the boolean section, at same level of the geo_location filter... And result was more or less the same. — Odín del Río Piñeiro, May 04 '16 at 21:42
Any memory (heap) pressure during these high latency periods? Increased number and/or duration for the old GCs? — Andrei Stefan, May 04 '16 at 21:49

Odín del Río Piñeiro · Answer 1 · 2016-05-28T11:59:59.813

Finally, I couldn't solve the issue performing the query like I described here...

I solved this creating 2 different document types, according to the TRUE/FALSE value of the related field.

With this approach, no CPU increase, no latency problems... And since ElasticSearch can search in multiple document types at a time, this separation wasn't cause major problems to my application code ;)

So now, I perform the same query in both cases but changing only the document type target:

POST /my_index/doc_type_with_true_value/_search
POST /my_index/doc_type_with_false_value/_search

{
  "from" : 0,
  "size" : 10,
  "query" : {
    "filtered" : {
      "query" : {
        "match_all" : { }
      },
      "filter" : {
        "and" : {
          "filters" : [ {
            "bool" : {
              "must" : [ {
                "range" : {
                  "_timestamp" : {
                    "from" : null,
                    "to" : "2016-05-04T15:12:00Z",
                    "include_lower" : true,
                    "include_upper" : false
                  }
                }
              } ]
            }
          }, {
            "geo_distance" : {
              "rounded_location" : [ -8.42, 42.24 ],
              "distance" : "300000m",
              "distance_type" : "plane",
              "optimize_bbox" : "indexed"
            }
          } ]
        }
      }
    }
  },
  "sort" : [ {
    "_geo_distance" : {
      "rounded_location" : [ {
        "lat" : 42.24,
        "lon" : -8.42
      } ],
      "unit" : "m"
    }
  }, {
    "date" : {
      "order" : "desc"
    }
  }, {
    "price" : {
      "order" : "asc"
    }
  } ]
}

Filter on boolean field increases latency and CPU

1 Answers1