9

I am trying to encrypt the Elasticsearch data. Are there any native methods to encrypt the data? I studied about Elasticsearch supporting dm-crypt, but there are no proper supporting documents on how it is being implemented. Also, my clients need free/opensource alternatives to Shield/X-Pack.

TIA.

sriramsm04
  • 343
  • 1
  • 7
  • 22
  • @WolfgangBlessen If you want to store encrypted data in your documents and search it at the same time, that won't be possible as ES cannot search encrypted data. Encryption at rest is only at the filesystem level. – Val Aug 08 '19 at 13:43
  • 2
    X-Pack is free now. Check this link for more https://www.elastic.co/what-is/open-x-pack – Sathishkumar Rakkiyasamy Aug 08 '19 at 21:02
  • Elasticsearch is meant for search and analytics. So storing sensitive data in ES is not the right choice in my opinion. You should add the data which are meant for search/analytics/visualization. – Sathishkumar Rakkiyasamy Aug 08 '19 at 21:04
  • Are you working for Biostar 2? *chuckles* – Anuga Aug 15 '19 at 10:40

1 Answers1

6

What is the supposed behaviour for encryption? Be able to search against encrypted data or just store some sensitive fields (e.g. PII) encrypted?

First of all, consider removing your sensitive data from ELK stack, as it isn't a reliable place to store it there. Detach it (just remove and store somewhere else or tokenize, if you want to have a link to it) or obfuscate before pushing to ELK.

As an alternative to performing encryption by means of Elasticsearch or its plugins, you can encrypt your data in the application that pushes data to ES in advance and just keep a couple of index fields (which are going to be used to search by) in plain text. E.g.

{ 
    "index_field" : "John Doe", // plain text to search by
    "address" : "s3_34$af78...", // encrypted
    "passport" : "3%75O9gfjdg4%...", // encrypted
    ...
}

That depends on your specific application, however in some cases solving encryption & key management questions would be much easier than looking for a particular solution e.g. for ES.

Vladimir Salin
  • 2,951
  • 2
  • 36
  • 49
  • So I understood the alternative solution by encrypting the data and then push to elasticsearch. What would happen if I had to pull Kibana into the picture? Will there be any challenge in visualizing the ES data if it is encrypted? – sriramsm04 Dec 04 '17 at 08:01
  • I am using this architecture. Logstash client pushes the logs to RabbitMQ. And in the ELK server, I have written a logstash configuration to pull the messages and then pass on to elasticsearch. What would be the ideal way to achieve data encryption at transit and rest as well? – sriramsm04 Dec 04 '17 at 08:03
  • You want to encrypt all of your logs and be able to search / visualize them at the same time? I think you'd better detach sensitive data from logs (e.g. use Logstash filter to detach, or even better, before pushing data to MQ) and put it away from ELK stack in a reliable data storage that supports encryption out of the box. The thing is, doing encryption on the fly will slow down entire process. At the same time, you'll need to apply encryption for RabbitMQ as well and protect through the whole chain. Kibana will struck with decryption in any way, unless you write your own plugin. – Vladimir Salin Dec 04 '17 at 08:09
  • So if I were to add security to the logs/messages I send, would it suffice if I enable SSL? Forgetting the encryption. I am a little new to ELK stack and trying to understand how it works. – sriramsm04 Dec 04 '17 at 09:10
  • SSL (or better TLS 1.2) is always good if you send sensitive data over the network. – Vladimir Salin Dec 04 '17 at 11:17
  • 1
    Okay I will look into implementing SSL or TLS! – sriramsm04 Dec 05 '17 at 05:33
  • @VladimirSalin Can you please give some directions for my question https://stackoverflow.com/questions/55501238/how-the-elasticsearch-works-when-the-data-is-encrypted-at-rest Actually I'm trying to store PII, but I would like to query the PII. – Muthaiah PL Apr 03 '19 at 21:05
  • In ELK stack, Elasticsearch(E) stores the data. Logstash(L) is the one which collects the data from different sources like logs, DB and etc. Kibana(K) is a UI tool to interact with ES and also visualize the data in the form of charts and dashboards. You can execute a query against ES in Kibana. Now we call it an Elastic Stack which includes Elasticsearch, Logstash, Kibana, X-Pack, and Beats. – Sathishkumar Rakkiyasamy Aug 08 '19 at 21:10
  • Answering to "What would be the ideal way to achieve data encryption at transit and rest as well": For in transit, simply use [Elastic Security](https://www.elastic.co/guide/en/elastic-stack-overview/current/elasticsearch-security.html) (i.e. SSL/TLS) which is now free, and for encryption at rest use [dm-crypt](https://discuss.elastic.co/t/how-should-i-encrypt-data-at-rest-with-elasticsearch/96), this will encrypt the filesystem, not your source documents though. – Val Aug 13 '19 at 04:18