Running background tasks in Meteor.js

Question

This is my scenario:

1. Scrape some data every X minutes from example.com
2. Insert it to Mongodb database
3. Subscribe for this data in Meteor App.

Because, currently I am not very good at Meteor this is what I am going to do:

1. Write scraper script for example.com in Python or PHP.
2. Run script every X minutes with cronjob.
3. Insert it to Mongodb.

Is it possible to do it completely with Meteor without using Python or PHP? How can I handle task that runs every X minutes?

score 4 · Answer 1 · edited May 23 '17 at 12:30

There are Cron like systems such as percolate:synced-cron for Meteor. There, you could register a job using Later.js syntax similar to this example taken from the percolate:synced-cron readme file:

SyncedCron.add({
  name: 'Crunch some important numbers for the marketing department',
  schedule: function(parser) {
    // parser is a later.parse object
    return parser.text('every 2 hours');
  }, 
  job: function() {
    var numbersCrunched = CrushSomeNumbers();
    return numbersCrunched;
  }
});

If you want to rely on an OS level cron job, you could just provide an HTTP endpoint in your Meteor.js application that you could then access through curl at the chosen time.

score 1 · Answer 2 · answered Jun 23 '18 at 11:12

I can suggest Steve Jobs, my new package for scheduling background jobs in Meteor.

You can use the register, replicate, and remove actions

// Register the job

Jobs.register({ 
    dataScraper: function (arg) {
        var data = getData()

        if (data) {
            this.replicate({
                in: {
                    minutes: 5
                }
            });

            this.remove(); // or, this.success(data)
        } else {
            this.reschedule({
                in: {
                    minutes: 5
                }
            })
        }
    }
})

// Schedule the job to run on Meteor startup
// `singular` ensures that there is only pending job with the same configuration

Meteor.startup(function () {
    Jobs.run("dataScraper", {
        in: {
            minutes: 5
        }, 
        singular: true
    })
})

Depending on your preference, you can store the result in the database, as part of the jobs history, or remove it entirely.

Running background tasks in Meteor.js

2 Answers2