59

I'm running an elastic search for a personal project on google-cloud and I use as a search index for my application. From the last 3 days, indexes are getting deleted mysteriously. I have no clue why, I looked at all my code for any delete index calls, also looked at logs. Still not able to figure it out. Any thoughts? How can I debug this?

[2020-07-24T00:00:27,451][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-1] [users_index_2/veGpdqbNQA2ZcnrrlGIA_Q] deleting index
[2020-07-24T00:00:27,766][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-1] [blobs_index_2/SiikUAE7Rb6gS3_UeIwElQ] deleting index
[2020-07-24T00:00:28,179][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [gk01juo8o3-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:28,776][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [28ds9nyf8x-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:29,328][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [hw2ktibxpl-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:29,929][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [va0pzk1hfi-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:30,461][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [ruwhw3jcx0-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:30,973][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [wx4gylb2jv-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:31,481][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [hbbmszdteo-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:31,993][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [1gi0x5277l-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:32,494][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [sotglodbi9-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:33,012][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [khvzsxctwr-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:33,550][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [hgrhythm3g-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:34,174][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [ejyucop7ag-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:34,715][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [n1bgkmqp8r-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:35,241][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [vsw49c4kpp-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:35,747][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [qrb5x89icr-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:36,261][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [pv8n84itx6-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:36,856][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [wnnwmylxvs-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:37,392][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [g5tw6w2tqb-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:37,889][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [u7tobv31o2-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:38,474][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [ufvizrnmez-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T00:00:38,946][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [0i9wszne7l-meow] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2020-07-24T01:30:00,001][INFO ][o.e.x.m.MlDailyMaintenanceService] [node-1] triggering scheduled [ML] maintenance tasks
[2020-07-24T01:30:00,002][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [node-1] Deleting expired data
[2020-07-24T01:30:00,010][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [node-1] Completed deletion of expired ML data
[2020-07-24T01:30:00,011][INFO ][o.e.x.m.MlDailyMaintenanceService] [node-1] Successfully completed [ML] maintenance tasks
[2020-07-24T01:30:00,039][INFO ][o.e.x.s.SnapshotRetentionTask] [node-1] starting SLM retention snapshot cleanup task
[2020-07-24T01:37:43,817][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [.kibana] creating index, cause [auto(bulk api)], templates [], shards [1]/[1], mappings []
Dharman
  • 30,962
  • 25
  • 85
  • 135
Pramod Shashidhara
  • 939
  • 2
  • 11
  • 17
  • It seems that some process deletes the index exactly at midnight... it looks like a cron job or something. Do you have anything installed in your crontab by any chance? – Val Jul 24 '20 at 05:39
  • 7
    Also plenty of new indexes named `xyz-meow` are created around the same time. Are these your indexes? – Val Jul 24 '20 at 05:40
  • Yeah, I can see that XYZ-meow's are not my indexes. They are some garbage indexes. Let me see if I have some crontab running. Thank You for your help. – Pramod Shashidhara Jul 24 '20 at 05:50
  • 3
    If those are not your indexes, it might be interesting to see what they contain, it might help to discover if your cluster has been compromised. I hope your cluster is not open to the world. – Val Jul 24 '20 at 05:52
  • I will see what ports are open and also allow only my IP address to connect to VM running the cluster. Is there anything else I can do? Thank You so much for your help, much appreciated. – Pramod Shashidhara Jul 24 '20 at 06:02
  • 4
    Since you're running 7.6, you should definitely enable xpack security which is free and set up proper user/roles. – Val Jul 24 '20 at 06:41
  • I see ML list in there. are you using Machine Learning in your stack? if so might be setting or behavior of ML? – LhasaDad Jul 24 '20 at 12:05
  • the other thing in there is SLM (snapshot lifecycle management) that might also have an affect on the indexes (it could be snapshotting to backup the current indexes. I would have to look at its settings to see if its possible that it is removing the indexes after backing them up. – LhasaDad Jul 24 '20 at 12:09
  • 7
    I’m voting to close this question because it belongs on Information Security Stack Exchange (https://security.stackexchange.com). – Ian Kemp Jul 27 '20 at 13:58
  • update your elasticsearch instance security group's inbound rules to allow only specific IPs you want to allow (for eg: your-laptop's-ip, other instances IP which use your ES) – Shubham Goel Sep 09 '20 at 06:52

6 Answers6

116

It looks like you are getting hit by a meow attack.

Hundreds of unsecured databases exposed on the public web are the target of an automated 'meow' attack that destroys data without any explanation.

The activity started recently by hitting Elasticsearch and MongoDB instances without leaving any explanation, or even a ransom note. Attacks then expanded to other database types and to file systems open on the web.

From this tweet, you can see that you are experiencing the same behavior seen by these attacks:

From the logs in MongoDB you can see it drops databases first then create new ones with $randomstring-meow

Please ensure that you are not using a default username and password for your DB and that your configuration is set up to avoid public-facing interactions. If you need to give access to your DB, use an API with key based auth, and only the bare minimum capabilities allowed.

Edit #1: You can obvserve the attacked databases here on shodan.io.

Edit #2: Some more advice for protecting from this (and other) attacks (from HackerNews user contrarianmop):

Also as a rule of thumb never ever expose anything but port 80 and 443 if hosting a webapp.

If you must expose services other than http/s then be sure to not leak its version, have it secured properly and always up to date. The user running such services should also be a non privileged user, the daemon chrooted, and the OS should have appropriate process and filesystem permissions in place.

Edit #3: An interesting theory as to why the attacker used the term "meow" is because cats like to drop (or knock) items from tables.

Shane Fontaine
  • 2,431
  • 3
  • 13
  • 23
  • 4
    This is good generic advice... except that ElasticSearch supports neither password nor api key authentication (in the free edition that most people use). – user5994461 Jul 26 '20 at 20:50
  • 14
    @user5994461 Then you probably shouldn’t use it on an internet facing server eh? – Aeolun Jul 26 '20 at 22:29
  • 19
    Since 6.8, security is available for free within the default distribution of elasticsearch. So the ability to protect from meow is free. Have a look at https://www.elastic.co/blog/how-to-prevent-elasticsearch-server-breach-securing-elasticsearch to see how to prevent an Elasticsearch server breach. – dadoonet Jul 27 '20 at 01:14
  • 5
    `term "meow" is because cats like to drop (or knock) items from tables` Hehe. I don't hear a lot about SQL servers where it's actually called "tables" getting meowed though. – powersource97 Jul 27 '20 at 04:23
  • 1
    One thing I find to be a real pain is when I can't access my SQL Server database directly with SQL Server Management Studio. Is that really so bad? The alternative would probably be to use Remote Desktop, which is also password protected. Is that really any safer? – Jonathan Wood Jul 27 '20 at 14:06
  • Related: https://security.stackexchange.com/questions/235318/what-is-the-meow-attack-and-how-can-i-guard-my-databases-against-it – tripleee Aug 03 '20 at 09:15
46

As answered by some people here, your cluster has been attacked by meow.

Since 6.8, security is available for free within the default distribution of elasticsearch. So the ability to protect from meow is free. Have a look at this blog post to see how to prevent an Elasticsearch server breach.

Update: Elastic also released a new blog post covering this specific Meow attack.

D Malan
  • 10,272
  • 3
  • 25
  • 50
dadoonet
  • 14,109
  • 3
  • 42
  • 49
27

You've been meowed:

https://www.bleepingcomputer.com/news/security/new-meow-attack-has-deleted-almost-4-000-unsecured-databases/

Double check all configurations (firewall, elastic search etc.) to ensure the instance is not publicly accessible and access to elastic search is secured (it must not have been before hand).

dijksterhuis
  • 1,225
  • 11
  • 25
13

Note: I just got your question featured on Hacker News, it's about to get a lot of visits and advice.

The elastic database is getting deleted by meow, a new operation scanning the internet for open databases and deleting them. See New "meow" attack has deleted almost 4,000 unsecured databases

There are two problems here:

  1. The database is fully exposed to the internet.

  2. The database is not protected by any form of authentication or access control.

What you need to do:

  1. Disconnect the database from the internet. If it's running on Google Cloud the instance shouldn't even have a public address, databases should be in an internal network (VPC).

  2. Restrict access to the database.

That last bit is unfortunately easier said that done. ElasticSearch doesn't support any form of access control in the free edition, assuming you have the free edition.

What you can do to restrict access is to firewall the instance tightly. This is easy enough to do with the firewall capabilities in Google/AWS/Azure. ElasticSearch typically listens to port 9200 for clients and 9300 or 9350 for replication. The only clients that need access are typically kibana to view logs and logstash/fluentd to ingest logs, that's only a couple of IP to allow traffic from.

If you are working in an enterprise, all production databases must be protected by authentication to satisfy a variety of regulations, so you have to pay up for an enterprise license and configure password or certificate authentication.

user5994461
  • 5,301
  • 1
  • 36
  • 57
  • 7
    Since 6.8, security is available for free within the default distribution of elasticsearch. So the ability to protect from meow is free. Have a look at https://www.elastic.co/blog/how-to-prevent-elasticsearch-server-breach-securing-elasticsearch to see how to prevent an Elasticsearch server breach. – dadoonet Jul 27 '20 at 01:12
9

Update: As mentioned in various answers and forums, indices were deleted due to meow attack and please follow Elasticsearch's official blog on how to freely secure your ES cluster from these attacks.

Old answer:

Try to see if by accident you have configured index life cycle management policy also if you are not creating the date based indices in your application.

Just had a look at your logs and these are clearly indicating that your ES is deleting the indices, Please see below lines from log which explains this.

[2020-07-24T00:00:27,451][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-1] [users_index_2/veGpdqbNQA2ZcnrrlGIA_Q] deleting index [2020-07-24T00:00:27,766][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node-1] [blobs_index_2/SiikUAE7Rb6gS3_UeIwElQ] deleting index

  • @PramodShashidhara if possible please provide the output of `_settings` in your ES cluster –  Jul 24 '20 at 05:07
  • ```{"blobs_index_2":{"settings":{"index":{"creation_date":"1595554664904","number_of_shards":"1","number_of_replicas":"1","uuid":"FwGpIco5Rnemm6jWEU8VXw","version":{"created":"7060299"},"provided_name":"blobs_index_2"}}}}``` – Pramod Shashidhara Jul 24 '20 at 05:10
  • What's ES? Can you please elaborate? – Pramod Shashidhara Jul 24 '20 at 05:51
  • ES short form of Elasticsearch, its clearly shown your in logs that your indices are getting deleted. –  Jul 24 '20 at 05:53
  • Thanks for your help, I will continue my research, seems like someone is deleting the indexes and creating some random indexes like xyz-meow's. – Pramod Shashidhara Jul 24 '20 at 06:03
6

If your ES instance was running on the free edition (and a version prior to 6.8), it likely got hit by a "meow attack" as ES free edition doesn't support any kind of access control in the free edition.

If it wasn't running on the free edition, or was locked behind a VPC of some sort, or wasn't exposed via 80 or 443, and it still dropped, then there are bigger issues.

Astra
  • 10,735
  • 3
  • 37
  • 41
Josh Brody
  • 5,153
  • 1
  • 14
  • 25
  • 9
    Since 6.8, security is available for free within the default distribution of elasticsearch. So the ability to protect from meow is free. Have a look at https://www.elastic.co/blog/how-to-prevent-elasticsearch-server-breach-securing-elasticsearch to see how to prevent an Elasticsearch server breach. – dadoonet Jul 27 '20 at 01:12