Continuous Query Processing with baseline indexing in Endeca

Question

I wanted to whether if it is possible to have Continuous Query Processing with baseline indexing in Endeca cluster environment. I have an Endeca central server and multiple Endeca agent only instances. If I run Baseline Indexing in the central server, it re-indexes the records in all the MDEX engines(all agents) simultaneously. I wanted to know if I can do it alternately on the agents so that while the records are getting indexed on one machine. The other system can still serve the user queries with the old Index. And the New Index kicks in only when we have completed indexing on all the machine. Continues Query Processing is available in partial update, is something like this possible for baseline indexing?

Answer 1

I think Dgraph's restartGroup property might help here.

The restartGroup property indicates the Dgraph's membership in a restart group. When applying a new index or configuration updates to a cluster of Dgraphs (or when updating a cluster of Dgraphs with a provisioning change such as a new or modified process argument), the Dgraph cluster object applies changes simultaneously to all Dgraphs in a restart group.

Answer 2

While the index is being built the old index will continue to run on the search servers. In order for a baseline update to be applied the mdex engine needs to be stopped and started.

Typically in a high availability environments there are multiple search servers responding to search requests. Should you apply the new index to all of them simultaneously, then there environment will stop responding to incoming queries for a minute or so while the restart is taking place.

If you wish to have zero downtime, you will need a short interval (typlically under a minute, depending on how many restarts take place in parallel) when some indexes are running the updated index and the others are not.

As mentioned this can be controlled with the Dgraph restart group