1

I got my Elasticsearch service disk storage space totally used, henceforth I had to increase the space, which worked. However, after that kibana stopped working with error on kibana URL as Tenant indices migration failed.

I've tried multiple links but most of them are pointing towards opendistro related settings which I am not able to access in this case as my configuration is over Amazon Elasticsearch service. One of the closest issue is About “Tenant indices migration failed” error in kibana

Below is the screenshot from my kibana: enter image description here

Any help will be highly appreciated. Thanks in Advance.

Shubham Namdeo
  • 1,845
  • 2
  • 24
  • 40
  • 1
    Seems there's still an open issue about it: https://github.com/opendistro-for-elasticsearch/security-kibana-plugin/issues/102 – Val Jun 09 '20 at 11:15
  • Thanks for the response @Val any possible fix or workaround for now? – Shubham Namdeo Jun 09 '20 at 11:20
  • 2
    Well, you need to ping the opendistro guys in that thread I guess. Or use a [real stack](https://www.elastic.co/aws-elasticsearch-service) :-) – Val Jun 09 '20 at 11:23
  • 1
    @Val , I also wanted to mention but you gave it a nice name uses a real stack :D – Amit Jun 09 '20 at 11:26
  • Yeah, it's more like a joke... but still, I can't help :-) – Val Jun 09 '20 at 11:33

2 Answers2

1

I've got a workaround for Amazon's Elasticsearch which worked for me for the time being, and may help others as well who are facing the same problem with this service in future until fix is finally added to AWS, I've added the step below:

  1. Once you encounter this error, wait for your nodes(data nodes, if dedicated nodes are present) to be active.
  2. Click on Edit domain in Elasticsearch service page on AWS.
  3. Change EBS storage size per node by 1GB (this will cause migrations to happen again and possibly fix indices migration error).
  4. Click Submit, and wait for domain to become active, once active try testing kibana url.
  5. If it does not work try changing EBS storage size per node by 1GB again.
  6. Last resort to be try changing Instance type and again after domain is active switch to previous type.

I've got this working solution by trial and error only. I am open for discussion on the same.

Shubham Namdeo
  • 1,845
  • 2
  • 24
  • 40
  • 1
    Great, glad to made it work. you can also add these steps on the open github issue – Amit Jun 10 '20 at 09:25
  • Certainly, I will add these there as well. – Shubham Namdeo Jun 10 '20 at 09:44
  • It didn't work for me. After change EBS storage size again and again the Domain Status is stuck in "Processing" for more than 24 hours. Status in Kibana is: "plugin:opendistro_security@7.1.1 = Tenant indices migration failed". Shards list (/_cat/shards) show two shards for ".opendistro_security" one "STARTED" and one "UNASSIGNED". – educoutinho Jun 24 '20 at 13:01
  • This should only be done once tenant is active after processing. If it is taking more than 24hours, there is something wrong with your domain, tou might need to try connecting with aws support. – Shubham Namdeo Jun 24 '20 at 13:07
1

I was running Kibana in Docker, so in my case I used this environment variable in my docker-compose:

environment:
  - OPENDISTRO_SECURITY_MULTITENANCY_ENABLED=false

Reference: https://github.com/opendistro-for-elasticsearch/security-kibana-plugin/issues/102

Tarasovych
  • 2,228
  • 3
  • 19
  • 51
  • Thanks for the solution, yes, this is one way to make this work, however if multi-tenancy is required as in my case then we can't make it false. – Shubham Namdeo Jul 09 '20 at 06:48