0

KB/ES version: 7.2
I am trying to build an ABAC security based Multi-Tenant system with X-Pack platinum license.
There is an Index which have 3 Client's data. The Admins who have access to all three Client's data, able to view the Kibana Dashboards properly.
However, Users and Admins with 2 out of 3 Clients' data access are unable to view the dashboard. Whenever they access the Dashboard the following Error pops up on Kibana.

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"unsupported_operation_exception","reason":"unsupported_operation_exception: null"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"unified_asset_processed","node":"MtkL9yGKQPyJZu3an3ICvw","reason":{"type":"unsupported_operation_exception","reason":"unsupported_operation_exception: null"}}]},"status":500}

Detailed Logs in ES can be seen like below:

org.elasticsearch.transport.RemoteTransportException: [elkxx-1][ip-address:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:306) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:114) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:335) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:360) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.2.0.jar:7.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.UnsupportedOperationException
        at org.elasticsearch.search.internal.ContextIndexSearcher$1.scorer(ContextIndexSearcher.java:156) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.xpack.core.security.authz.accesscontrol.SecurityIndexSearcherWrapper$IndexSearcherWrapper.search(SecurityIndexSearcherWrapper.java:164) ~[?:?]
        at org.apache.lucene.search.XIndexSearcher.search(XIndexSearcher.java:44) ~[elasticsearch-7.2.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
        at org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:177) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:443) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:271) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:114) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:335) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:360) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) [elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:835) ~[?:?]

How to resolve this error?

Kibana request posted to ES: 

Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[xxxx_xxxx_processed], indicesOptions=IndicesOptions[ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='1569247377788', requestCache=false, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=32, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":0,"timeout":"30000ms","query":{"bool":{"must":[{"range":{"CREATE_DATE":{"from":"2019-02-24T06:00:49.223Z","to":"2019-09-24T06:00:49.223Z","include_lower":true,"include_upper":true,"format":"strict_date_optional_time","boost":1.0}}}],"filter":[{"match_all":{"boost":1.0}},{"match_all":{"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"_source":{"includes":[],"excludes":[]},"stored_fields":"*","docvalue_fields":[{"field":"WOW_CREATE_DTTM","format":"date_time"},{"field":"CREATE_DATE","format":"date_time"},{"field":"INSTALL_DATE","format":"date_time"},{"field":"LASTMODIFIEDTIME","format":"date_time"},{"field":"LAST_REFRESH_DTTM","format":"date_time"},{"field":"PRODDATE","format":"date_time"}],"script_fields":{},"track_total_hits":2147483647,"aggregations":{"2":{"terms":{"field":"OSNAME2","missing":"__missing__","size":50,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}]
Soumendra
  • 1,518
  • 3
  • 27
  • 54
  • 1
    Are you trying to view the data from Kibana? Can you paste the request sent to ES, with the method, path and body? If this is a kibana dashboard that fails, I would recommend trying to view each visualization one by one, see which visualisation fail, pinpointing the issue – Opster Elasticsearch Expert Sep 24 '19 at 07:54
  • Another option is that the index you are searching on was created in an old ES version, with incompatible mapping. see issue and workaround here: https://github.com/elastic/kibana/issues/13950 – Opster Elasticsearch Expert Sep 24 '19 at 07:57
  • Yes, I am viewing from Kibana. Added the KB --> ES request details on question. I could not figure out how to see each visualization seprately. Evey time I give the visualization URL, the dashboard drop down opens up. – Soumendra Sep 24 '19 at 10:55
  • looking at your query, it looks like a single query and not a dashboard (or a dashboard containing only a single visualisation). I can guess that the issue involves querying a field with the wrong/different mapping - like a numeric field being treated like a date. a good way to debug this would be to take your query (the json query you added to the question) and run it directly to ES, using postman/curl. the query should fail, just like in kibana, and now start removing parts of the query, until you find a simpler query that doesn't fail - this way you can find the bad part – Opster Elasticsearch Expert Sep 24 '19 at 11:03

1 Answers1

0

On what ES version was the index created on? might be that the index was created in ES 5 or less? that might be the case, as mapping types have changed and query syntax might not fit. see this issue for a similar example: github.com/elastic/kibana/issues/13950

Opster Elasticsearch Expert
  • 1,837
  • 1
  • 15
  • 18
  • ES index was created on the same version, 7.2.0 So this is not the case. – Soumendra Sep 24 '19 at 11:26
  • it might be security related - looks like the user you are using don't have permissions to run the search on this specific index. did you set up role base access or users permissions in kibana? – Opster Elasticsearch Expert Sep 24 '19 at 11:33
  • Yes, Both RBAC and ABAC enabled. – Soumendra Sep 24 '19 at 11:47
  • It sounds like one of the visualisations is accessing data which it doesn't have authorisation for. either update the rules to allow it, or open the visualisations one by one in the visualisations tab, and see which one/s fails and remove them from the dashboard – Opster Elasticsearch Expert Sep 24 '19 at 13:59