Meteor - Display feed of multiple collections

Question

I'd like to display a feed of items from various collections, sorted by creation date but i'm facing some problems.

Desired result

Items from all collections should be mixed on the same page, sorted by date
The display must stay reactive
I use jQuery Isotope for the layout
The result set is limited (10 first items for example) and a 'load more' button can be clicked to expand the query limit

I can't limit the result set correctly

As each collection must be queried separately, the best I can do is get the 10 first items for each collection.

In the illustration below, the desired result set is in red and the green outlines are the ones I get.

Currently, I fetch the three cursors, merge in one array, sort and keep the first ten.

Loading more content

When the 'load more' button is clicked, a reactive variable is updated, which is used in the subscription to limit the results for each queries.

That logic of loading too much data every time feels very wrong. What would be the performance impact for a page that would query 3000 elements, sort them in an array, and display only the 1000 first.

I may overlooked a very simple way to do it. I'm sure that some brilliant minds here can help me find a better approach.

Some code

Limited to two collections to make it a bit more concise

// Server
Meteor.publish('allContent', function(limit) {
    return [
        Posts.find({}, {limit: limit}),
        Articles.find({}, {limit: limit})
    ];
});

// Client
Template.home.onCreated(function () {
    var instance = this;

    // initialize the reactive variables
    instance.loaded = new ReactiveVar(0);
    instance.limit = new ReactiveVar(10);

    instance.autorun(function () {

        // get the limit
        var limit = instance.limit.get();

        // subscribe to the posts publication
        var subscription = instance.subscribe('allContent', limit);

        // if subscription is ready, set limit to newLimit
        if (subscription.ready()) {
            instance.loaded.set(limit);
        }
    });

    instance.content = function() {
        var llimit = instance.loaded.get();

        var posts_cursor = Posts.find({}, {limit: llimit});
        var articles_cursor = Articles.find({}, {limit: llimit});

        var posts_docs = posts_cursor.fetch();
        var articles_docs = articles_cursor.fetch();

        var docs = posts_docs.concat(articles_docs);
        var mix = _.sortBy(docs, function(doc) {return doc.created_at;}).reverse().slice(0,Session.get('limit'));

        // Pass the cursors to observe them and re-layout isotope when necessary
        return {items: mix, cursors: [services_cursor, posts_cursor]};
    }

score 2 · Answer 1 · edited May 23 '17 at 11:52

2

Here is a solution that fits into your requirements, but it will put more load on the server.

You need to do determine what is the limit for each collection in your publication, and publish them accordingly. Let's say your limit is set to 10.

Find and fetch the 10 first items of each collection. Note that you can sort them by date in your mongo request you are using at this step. See here and here

Merge them in an array but keep track of their origin. For instance you could do this:

var mergedResults = [];
_.each( TenLatestPostArray, function( post){
      //we choose the id 1 for the posts
      mergedResults.push({post.createdAt, 1})        
 })
 _.each( TenLatestArticlesArray, function( article){
      //we choose the id 2 for the articles
      mergedResults.push({article.createdAt, 2})        
 })
 //same for your third collection

You sort your mergedResults by date, and slice the 10 first results
You pluck the ids from the array and count how many items in the resulting array are from collection 1,2,and 3

You publish it following each collection specific limits:

Meteor.publish('allContent', function(limit) {
  return [
    Posts.find({}, {limit: postsLimit}),
    Articles.find({}, {limit: articlesLimit})
  ];
});

edited May 23 '17 at 11:52

Community

1
1

answered Dec 17 '15 at 12:57

Billybobbonnet

3,156
4
23
49

It looks like the reactivity would be lost because the limit argument is not used anymore in the publication. And as you said, the performance hit is worse. There's now the double amount of queries. – Manuszep Dec 17 '15 at 16:11
Yes it is not an ideal solution. You could use `cursor.observe()` to update the publication but it makes it even more overkill. FYI, the base logic is here: http://stackoverflow.com/a/30813050/3793161 – Billybobbonnet Dec 17 '15 at 16:12
alternatively, you could also use https://github.com/percolatestudio/publish-counts and resubscribe every time your collection count changes. Still an overkill solution, not as reliable as the former one – Billybobbonnet Dec 17 '15 at 16:14
Well I already use cursor.observe() to relayout isotope but I was hoping to keep things as simple as possible. With a relational DB, it's so common I can't believe there's no good way to achieve this – Manuszep Dec 17 '15 at 16:16
Yes, aggregation + reactivity is not an easy mix. I usually solve these issues by merging the data client side, as you do, but I agree it is not satisfying. – Billybobbonnet Dec 17 '15 at 16:17
Even worst and dirtier, you could resubscribe periodically, using a `setInterval()`. Worth mentioning, not worth making, I would say. – Billybobbonnet Dec 17 '15 at 16:21

Meteor - Display feed of multiple collections

Desired result

I can't limit the result set correctly

Loading more content

Some code

1 Answers1