1

I'm working on a filtered live search module with Meteor.js.

Usecase & problem:

A user wants to do a search through all the users to find friends. But I cannot afford for each user to ask the complete users collection. The user filter the search using checkboxes. I'd like to subscribe to the matched users. What is the best way to do it ?

I guess it would be better to create the query client-side, then send it the the method to get back the desired set of users. But, I wonder : when the filtering criteria changes, does the new subscription erase all of the old one ? Because, if I do a first search which return me [usr1, usr3, usr5], and after that a search that return me [usr2, usr4], the best would be to keep the first set and simply add the new one to it on the client-side suscribed collection.

And, in addition, if then I do a third research wich should return me [usr1, usr3, usr2, usr4], the autorunned subscription would not send me anything as I already have the whole result set in my collection.

The goal is to spare processing and data transfer from the server.

I have some ideas, but I haven't coded enough of it yet to share it in a easily comprehensive way.

How would you advice me to do to be the more relevant possible in term of time and performance saving ?

Thanks you all.

David

David Panart
  • 656
  • 6
  • 20
  • Creating a query _client-side_, then executing it on the server? It's literally asking for injections. Also, your question might be seen as too broad because you ask for implementation advices. – Kyll May 29 '15 at 18:33
  • I'm working on the implementation. I'll post code asap. But they are question that you may be could answer, like do a new subscription with new parameters erase all of the old one, or does it merge it in the client-side collection ? – David Panart May 29 '15 at 18:44
  • Making multiple subscriptions will add to your existing data. Here's a good [link](https://www.discovermeteor.com/blog/understanding-meteor-publications-and-subscriptions/) – phunktrain May 29 '15 at 21:07
  • Yes, iI nkew that, I read this very good article ;) But it doesn't explain if, s the server keeps a save of every users available collection / documents, when it proceed to another transfer to the client, does the server look at which are already on the clietn to save processing, or does it recompute even the already existing documents and resend them with the new ones ? Thanks though – David Panart May 29 '15 at 21:59

2 Answers2

1

It depends on your application, but you'll probably send a non-empty string to a publisher which uses that string to search the users collection for matching names. For example:

Meteor.publish('usersByName', function(search) {
  check(search, String);

  // make sure the user is logged in and that search is sufficiently long
  if (!(this.userId && search.length > 2))
    return [];

  // search by case insensitive regular expression
  var selector = {username: new RegExp(search, 'i')};

  // only publish the necessary fields
  var options = {fields: {username: 1}};

  return Meteor.users.find(selector, options);
});

Also see common mistakes for why we limit the fields.

performance

Meteor is clever enough to keep track of the current document set that each client has for each publisher. When the publisher reruns, it knows to only send the difference between the sets. So the situation you described above is already taken care of for you.

  1. If you were subscribed for users: 1,2,3
  2. Then you restarted the subscription for users 2,3,4
  3. The server would send a removed message for 1 and an added message for 4.

Note this will not happen if you stopped the subscription prior to rerunning it.

To my knowledge, there isn't a way to avoid removed messages when modifying the parameters for a single subscription. I can think of two possible (but tricky) alternatives:

  1. Accumulate the intersection of all prior search queries and use that when subscribing. For example, if a user searched for {height: 5} and then searched for {eyes: 'blue'} you could subscribe with {height: 5, eyes: 'blue'}. This may be hard to implement on the client, but it should accomplish what you want with the minimum network traffic.

  2. Accumulate active subscriptions. Rather than modifying the existing subscription each time the user modifies the search, start a new subscription for the new set of documents, and push the subscription handle to an array. When the template is destroyed, you'll need to iterate through all of the handles and call stop() on them. This should work, but it will consume more resources (both network and server memory + CPU).

Before attempting either of these solutions, I'd recommend benchmarking the worst case scenario without using them. My main concern is that without fairly tight controls, you could end up publishing the entire users collection after successive searches.

David Weldon
  • 63,632
  • 11
  • 148
  • 146
  • Thanks for the explanation. But that's my point : if the server sends me `removed` message for usr1, it will be deleted from the client, no ? I'd want it to be kept, as it might be re-added with a new parameter checked / unchecked in filters :/ – David Panart May 30 '15 at 14:56
0

If you want to go easy on your server, you'll want to send as little data to the client as possible. That means every document you send to the client that is NOT a friend is waste. So let's eliminate all that waste.

Collect your filters (eg filters = {sex: 'Male', state: 'Oregon'}). Then call a method to search based on your filter (eg Users.find(filters). Additionally, you can run your own proprietary ranking algorithm to determine the % chance that a person is a friend. Maybe base it off of distance from ip address (or from phone GPS history), mutual friends, etc. This will pay dividends in efficiency in a bit. Index things like GPS coords or other highly unique attributes, maybe try out composite indexes. But remember more indexes means slower writes.

Now you've got a cursor with all possible friends, ranked from most likely to least likely.

Next, change your subscription to match those friends, but put a limit:20 on there. Also, only send over the fields you need. That way, if a user wants to skip this step, you only wasted sending 20 partial docs over the wire. Then, have an infinite scroll or 'load more' button the user can click. When they load more, it's an additive subscription, so it's not resending duplicate info. Discover Meteor describes this pattern in great detail, so I won't.

After a few clicks/scrolls, the user won't find any more friends (because you were smart & sorted them) so they will stop trying & move on to the next step. If you returned 200 possible friends & they stop trying after 60, you just saved 140 docs from going through the pipeline. There's your efficiency.

Matt K
  • 4,813
  • 4
  • 22
  • 35