I am deploying my container to AWS EKS which is running elasticsearch. I need to pick up a storage service to store all the data. I searched a few articles and people mentioned EBS. Is this a good storage for data source like elasticsearch? Is there any better solution?
I know AWS provides hosted elasticsearch solution but I don't want to stick on AWS. By using EKS, it is easy for me to migrate to other cloud later.
AWS provides a managed Elasticsearch service called Amazon Elasticsearch Service which is based on the OpenDistro Elasticsearch. Which means you are not tied into AWS. AWS just manages the cluster on your behalf. But because it is an AWS managed service it supports additional services such as Amazon CloudWatch and Amazon S3. If you are using Amazon Elasticsearch service you can store data on S3. See the blog post "Use Amazon S3 to Store a Single Amazon Elasticsearch Service Index". Amazon Elasticsearch service has a new cost effective way to store large amounts of read-only data by moving the data from EBS to S3. This new service is known as Ultrawarm. More information here.
To answer your question.
What do you want to store? As there are a few options. But as you mentioned EBS (Amazon Elastic Block Storage). I assume you want to store the Elastic Search data. In this case I would recommend you look at Amazon Elastic File System (EFS).
EBS is block storage bound to an EC2 instance. In your case the EC2 instance running as the EKS node. Generally EBS is bound to one EC2 instance only. While it does support multiple EC2 instances too with Amazon EBS Multi-Attach. But think of EBS as a hard drive.
On the other hand EFS is network storage that is scalable and can be linked to multiple containers (EKS, ECS, and Fargate), EC2 instances, and Lambda functions.
The following blog post explains the new feature - AWS Fargate for Amazon EKS now supports EFS. In summary AWS released a driver called the Amazon EFS CSI driver. The documentation is here. You can use this driver to mount and EFS volume within your container. Because it is network storage, you can share that volume with other instances.
The EKS Workshop has a section on how to link EFS to EKS. You can follow the workshop here.
Alternatively please visit the support response for "How do I use persistent storage in Amazon EKS"
Since you are using Fargate for your EKS, I think good option would be to use Amazon Elastic File System which would persist your data even if you termiante your EKS cluster. What's more the data on the EFS could be used at the same time from ECS tasks, EC2 instances or even Lambda function.
EFS can be used with EKS Fargate as explained in the AWS blog:
EFS provides a simple, scalable, and fully managed shared file system for use with AWS cloud services, and can also help Kubernetes applications be highly available because all data written to EFS is written to multiple AWS Availability Zones. EFS is built for on-demand petabyte growth without application interruption, and it automatically grows and shrinks as files are added and removed, eliminating the need to provision and manage capacity to accommodate growth.
