0

I am building a social network and I want to use ElasticSearch to search for the users. Apart from creating the ElasticSearch indexes, I need to store the user information in database to login and this kind of things, isn't that redundant?

Aliuk
  • 1,249
  • 2
  • 17
  • 32
  • But the question is: do I have to create an index each time I store a user in the database? – Aliuk Aug 22 '16 at 10:46
  • @Aliuk - you need to index that particular user in elasticsearch. Anytime you add, update, delete data from your main store you need to update the same in elasticsearch too. – Kedar Parikh Aug 22 '16 at 10:48
  • How do I automate the process outside from the application? I'm sorry, I'm a beginner in ElasticSearch. – Aliuk Aug 22 '16 at 10:49
  • Take a look @ this answer: http://stackoverflow.com/questions/24341441/automatically-syncing-elasticsearch-with-sql – Kedar Parikh Aug 22 '16 at 10:50
  • "How do i automate the process outside from the applications?".Asking for advice/help like this, should be helped with your tech stack, primary database you are using, API layers. – user3775217 Aug 22 '16 at 11:56
  • I think I'm going to do it with a schedule that executes every 15 minutes or so. I don't think it is so important to have the user index updated immediatly. – Aliuk Aug 22 '16 at 12:06

1 Answers1

1

First: think about why you need a database if the data is already in Elasticsearch. Is that really necessary.

Second: if you need data in more than one system, define which is the leading system. It will be guaranteed to always hold the latest data. The other ones may lag.

Third: there are cases when Elasticsearch as a searchable data lake is just wonderful and sufficient. Unless you need transactional logic, you may be fine without a database. However, if you have a requirement for special features covered only by a certain database, then you need both.

In my experience, that's mostly one of the following cases:

  • you need a transactional database (like MySQL, MariaDB, Oracle, ...) to manage data records, and Elasticsearch is the fast retrieval engine for it (example: directory services, phone book)

  • you need an analytic database (e.g., Exasol) to run more complex retrievals and analytics on hot data, while Elasticsearch is used for warm/cold data with a simpler attribute-driven retrieval semantics

  • you need a high-performance database (e.g., Exasol, Apache Cassandra) for the ingestion of large-volume data or large-scale operations, and Elasticsearch is the longer-term data lake in the background.

The case in which structured and unstructured data are required does not necessarily call for a database as you can easily abuse Elastisearch as a database.

So, your question is really too general and I would recommend to seek professionals helping you with setting this up before you really build that social network. Otherwise, you may find yourself facing major business problems if your platform does not mirror business requirements properly.

j94305
  • 69
  • 1