Meteor 0.9.1.1 - Populating from json endpoint to Server side collections

Question

I am writing a package as part of a small application I am working on and one thing I need to do is fetch json data from an endpoint and populate it to a Server side collection.

I have been receiving error messages telling me I need to put by server side collection update function into a Fiber, or Meteor.bindEnvironment, or Meteor._callAsync.

I am puzzled, because there are no clear and concise explanations telling me what these do exactly, what they are, if and when they are being deprecated or whether or not their use is good practice.

Here is a look at what is important inside my package file

api.addFiles([
    '_src/collections.js'
], 'server');

A bit of psuedo code:

1) Set up a list of Mongo.Collection items

2) Populate these using a function I have written called httpFetch() and run this for each collection, returning a resolved promise if the fetch was successful.

3) Call this httpFetch function inside an underscore each() loop, going through each collection I have, fetching the json data, and attempting to insert it to the Server side Mongo DB.

Collections.js looks like what is below. Wrapping the insert function in a Fiber seems to repress the error message but no data is being inserted to the DB.

/**

* Server side component makes requests to a remote * endpoint to populate server side Mongo Collections. * * @class Server * @static */ Server = {

Fiber: Npm.require('fibers'),

/**
 *  Collections to be populated with content
 *  
 *  @property Collections
 *  @type {Object}
 */
Collections: {
    staticContent:  new Mongo.Collection('staticContent'),
    pages:          new Mongo.Collection('pages'),
    projects:       new Mongo.Collection('projects'),
    categories:     new Mongo.Collection('categories'),
    formations:     new Mongo.Collection('formations')  
},

/**
 *  Server side base url for making HTTP calls
 *  
 *  @property baseURL
 *  @type {String}
 */
baseURL: 'http://localhost:3000',

/**
 *  Function to update all server side collections
 *
 *  @method updateCollections()
 *  @return {Object} - a resolved or rejected promise
 */
updateCollections: function() {

    var deferred = Q.defer(),
        self = this,
        url = '',
        collectionsUpdated = 0;

    _.each(self.Collections, function(collection) {

        // collection.remove();
        url = self.baseURL + '/content/' + collection._name + '.json';

        self.httpFetch(url).then(function(result) {

            jsonData = EJSON.parse(result.data);

            _.each(jsonData.items, function(item) {
                console.log('inserting item with id ', item.id);
                self.Fiber(function() {
                    collection.update({testID: "Some random data"}
                });
            });

            deferred.resolve({
                status: 'ok',
                message: 'Collection updated from url: ' + url
            });

        }).fail(function(error) {
            return deferred.reject({
                status: 'error',
                message: 'Could not update collection: ' + collection._name,
                data: error
            });
        });

    });

    return deferred.promise;
},

/**
 *  Function to load an endpoint from a given url
 *
 *  @method httpFetch()
 *  @param  {String} url
 *  @return {Object} - A resolved promise if the data was
 *                     received or a rejected promise.
 */
httpFetch: function(url) {

    var deferred = Q.defer();

    HTTP.call(
        'GET',
        url,
        function(error, result) {
            if(error) {
                deferred.reject({
                    status: 'error',
                    data: error
                });
            }
            else {
                deferred.resolve({
                    status: 'ok',
                    data: result.content
                }); 
            }
        }
    );
    return deferred.promise;
}

};

I am still really stuck on this problem, and from what I have tried before from reading other posts, I still can't seem to figure out the 'best practice' way of getting this working, or getting it working at all.

There are plenty of suggestions from 2011/2012 but I would be reluctant to use them, since Meteor is in constant flux and even a minor update can break quite a lot of things.

Thanks

score 1 · Accepted Answer · answered Sep 15 '14 at 08:20

Good news : the solution is actually much simpler than all the code you've written so far.

From what I've grasped, you wrote an httpFetch function which is using the asynchronous version of HTTP.get decorated with promises. Then you are trying to run your collection update in a new Fiber because async HTTP.get called introduced a callback continued by the use of promise then.

What you need to do in the first place is using the SYNCHRONOUS version of HTTP.get which is available on the server, this will allow you to write this type of code :

updateCollections:function(){
  // we are inside a Meteor.method so this code is running inside its own Fiber
  _.each(self.Collections, function(collection) {
    var url=// whatever
    // sync HTTP.get : we get the result right away (from a
    // code writing perspective)
    var result=HTTP.get(url);
    // we got our result and we are still in the method Fiber : we can now
    // safely call collection.update without the need to worry about Fiber stuff
  });

You should read carefully the docs about the HTTP module : http://docs.meteor.com/#http_call

score 0 · Answer 2 · answered Sep 15 '14 at 11:19

I now have this working. It appears the problem was with my httpFetch function returning a promise, which was giving rise to the error:

"Error: Meteor code must always run within a Fiber. Try wrapping callbacks that you pass to non-Meteor libraries with Meteor.bindEnvironment."

I changed this httpFetch function to run a callback when HTTP.get() had called with success or error.

Inside this callback is the code to parse the fetched data and insert it into my collections, and this is the crucial part that is now working.

Below is the amended Collections.js file with comments to explain everything.

Server = {

/**
 *  Collections to be populated with content
 *  
 *  @property Collections
 *  @type {Object}
 */
Collections: {
    staticContent:  new Mongo.Collection('staticContent'),
    pages:          new Mongo.Collection('pages'),
    projects:       new Mongo.Collection('projects'),
    categories:     new Mongo.Collection('categories'),
    formations:     new Mongo.Collection('formations')  
},

/**
 *  Server side base url for making HTTP calls
 *  
 *  @property baseURL
 *  @type {String}
 */
baseURL: 'http://localhost:3000',

/**
 *  Function to update all server side collections
 *
 *  @method updateCollections()
 *  @return {Object} - a resolved or rejected promise
 */
updateCollections: function() {

    var deferred = Q.defer(),
        self = this,
        collectionsUpdated = 0;

    /**
     *  Loop through each collection, fetching its data from the json 
     *  endpoint.
     */
    _.each(self.Collections, function(collection) {

        /**
         *  Clear out old collection data
         */
        collection.remove({});

        /**
         *  URL endpoint containing json data. Note the name of the collection
         *  is also the name of the json file. They need to match.
         */
        var url = self.baseURL + '/content/' + collection._name + '.json';

        /**
         *  Make Meteor HTTP Get using the function below.
         */
        self.httpFetch(url, function(err, res) {

            if(err) {
                /**
                 *  Reject promise if there was an error
                 */
                deferred.reject({
                    status: 'error',
                    message: 'Error fetching content for url ' + url,
                    data: err
                });
            }
            else {
                /**
                 *  Populate fetched data from json endpoint
                 */
                var jsonData = res.content;
                    data = EJSON.parse(res.content);

                /**
                 *  Pick out and insert each item into its collection
                 */
                _.each(data.items, function(item) {
                    collection.insert(item);
                });

                collectionsUpdated++;

            }

            if(collectionsUpdated === _.size(self.Collections)) {

                /**
                 *  When we have updated all collections, resovle the promise
                 */
                deferred.resolve({
                    status: 'ok',
                    message: 'All collections updated',
                    data: {
                        collections: self.Collections,
                        count: collectionsUpdated
                    }
                });
            }
        });

    });

    /**
     *  Return the promise
     */

    return deferred.promise;
},

/**
 *  Function to load an endpoint from a given url
 *
 *  @method httpFetch()
 *  @param  {String} url
 *  @param  {Function} cb - Callback in the event of an error
 *  @return undefined
 */
httpFetch: function(url, cb) {

    var res = HTTP.get(
        url, 
        function(error, result) {
            cb(error, result);
        }
    );
}

};

Meteor 0.9.1.1 - Populating from json endpoint to Server side collections

2 Answers2