how to implement saved-searches scenario

Question

what is saved-search?

Save is the mechanism users don't find their desired results in advanced search and just push "Save My Search Criteria bottom" and we save the search criteria and when corresponding data post to website we will inform the user "hey user, the item(s) you were looking for exists now come and visit it".

Saved Searches is useful for sites with complex search options, or sites where users may want to revisit or share dynamic sets of search results.

we have advanced search and don't need to implement new search, what we require is a good performance scenario to achieve saved-search mechanism.

we have a website that users post about 120,000 posts per day into the website and we are going to implement SAVED SEARCH scenario(something like this what https://www.gumtree.com/ do), it means users using advanced search but they don't find their desired content and just want to save the search criteria and if there will be any results in the website we inform them with notification.

We are using Elastic search and Mysql in our Website.We still, haven't implement anything and just thinking about it to find good solution which can handle high rate of date, in other hand **the problem is the scale of work, because we have a lot of posts per day and also we guess users use this feature a lot, So we are looking for good scenario which could handle this scale of work easy with high performance.

suggested solutions but not the best

• one quick solution is we save the saved-searches in saved-search-index in Elastic then run a cronjob that for all saved-searches items get results from posts-index- Elastic and if there is any result push a record into the RabbitMq to notify the equivalent user.

• on user post an item into the website we check it with exists saved-searches in saved-search-index in Elastic and if matched we put a record into the RabbitMq,( the main problem of this method is it could be matched with a huge number of saved-searches in every post inserted into the website).

My big concern is about scale and performance, I'll appreciate sharing your experiences and ideas about this problem with me.

My estimation about the scale

• Expire date of saved-search is three month
• at least 200,000 Saved-search Per day
• So we have 9,000,000 active Records

I'll appreciate if you share your mind with me

*just FYI** - we also have RabbitMQ for our queue jobs - our ES servers are good enough with 64GB RAM

Answer 1

Cron job - No. Continual job - yes.

Why? As things scale, or as activity spikes, cron jobs become problematical. If the cron job for 09:00 runs too long, it will compete for resources with the 10:00 instance; this can cascade into a disaster.

At the other side, if a cron job finishes 'early', then the activity oscillates between "busy" (the cron job is doing stuff) and "not busy" (cron has finished, and not time for next invocation).

So, instead, I suggest a job that continually runs through all the "stored queries", doing them one at a time. When it finishes the list, is simply starts over. This completely eliminates my complaints about cron, and provides an automatic "elasticity" to handle busy/not-busy times -- the scan will slow down or speed up accordingly.

When the job finishes, the list, it starts over on the list. That is, it runs 'forever'. (You could use a simple cron job as a 'keep-alive' monitor that restarts it if it crashes.)

OK, "one job" re-searching "one at a time" is probably not best. But I disagree with using a queuing mechanism. Instead, I would have a small number of processes, each acting on some chunk of the stored queries. There are many ways: grab-and-lock; gimme a hundred to work on; modulo N; etc. Each has pros and cons.

Answer 2

Because you are already using Elasticsearch and you have confirmed that you are creating something like Google Alerts, the most straightforward solution would be Elasticsearch Percolator.

From the official documentation, Percolator is useful when:

You run a price alerting platform which allows price-savvy customers to specify a rule like "I am interested in buying a specific electronic gadget and I want to be notified if the price of gadget falls below $X from any vendor within the next month". In this case you can scrape vendor prices, push them into Elasticsearch and use its reverse-search (Percolator) capability to match price movements against customer queries and eventually push the alerts out to the customer once matches are found.

I can't say much when it comes to performance, because you did not provide any example of your queries but mostly because my findings are inconsistent.

According to this post (https://www.elastic.co/blog/elasticsearch-queries-or-term-queries-are-really-fast), Elasticsearch queries should be capable of reaching 30,000 queries/second. However, this unanswered question (Elasticsearch percolate performance) reported a painfully slow 200 queries/second on a 16 CPU server.

With no additional information I can only guess that the cause is configuration problems, so I think you'll have to try a bunch of different configurations to get the best possible performance. Good luck!

Answer 3

This answer was written without a true understanding of the implications of a "saved search". I leave it here as discussion of a related problem, but not as a "saved search" solution. -- Rick James

If you are saving only the "query", I don't see a problem. I will assume you are saving both the query and the "resultset"...

One "saved search" per second? 2.4M rows? Simply rerun the search when needed. The system should be able to handle that small a load.

Since the data is changing, the resultset will become outdated soon? How soon? That is, saving the resultset needs to be purged rather quickly. Surely the data is not so static that you can wait a month. Maybe an hour?

Actually saving the resultset and being able to replay it involves (1) complexity in your code, (2) overhead in caching, I/O, etc, etc.

What is the average number of times that the user will look at the same search? Because of the overhead I just mentioned, I suspect the average number of times needs to be more than 2 to justify the overhead.

Bottomline... This smells like "premature optimization". I recommend

1. Build the site without saving resultsets.
2. Stress test it to see when it will break.
3. Work on optimizing the slow parts.

As for RabbitMQ -- "Don't queue it, just do it". The cost of queuing and dequeuing is (1) increased latency for the user and (2) increased overhead on system. The benefit (at your medium scale) is minimal.

If you do hit scaling problems, consider

• Move clients off to another server -- away from the database. This will give you some scaling, but not 2x. To go farther...
• Use replication: One Master + many readonly Slaves -- and do the queries on the Slaves. This gives you virtually unlimited scaling in the database.
• Have multiple web servers -- virtually unlimited scaling in this part.

Answer 4

I don't understand why you want to use saved-search... First: you should optimize service, so as to use as little as possible the saved-search.

Have you done anything with the ES server? (What can you afford), so:

1. Have you optimized elasticearch server? By default, it uses 1GB of RAM. The best solution is to give him half the machine RAM, but no more than 16GB (if I'm remember. Check doc)
2. How powerful is the ES machine? He likes core instead of MHZ.
3. How many ES nodes do you have? You can always add another machine to get the results faster.
4. In my case (ES 2.4), the server slows down after a few days, so I restart it once a day.

And next:

1. Why do you want to fire up tasks every half hour? If you already use cron, fire then every minute, and you indicate that the query is running. With the other the post you have a better solution and an explanation.
2. Why do you separate the result from the query?
3. Remember to standardize the query to change the order of the parameters, not to force a new query.
4. Why do you want to use MySQL to store results? The better document-type database, like Elasticsearch xD.

I propose you:

1. Optimize ES structure - choose right tokenisers for fields.
2. Use asynchronous loading of results - eg WebSocket + Node.js