3

For a site that wants the "realtime" "reactivity" that Meteor provides on the frontend app, and have a job processing backend (something like Kue), clearly the frontend app benefits from Meteor. The backend processing does not need the reactivity of Meteor, except in realtime reporting in Admin UIs.

I do understand that Meteor is a full stack and handles both frontend and backend. When I state frontend in my question, its everything related to serving the UI for the users, so frontend app will include the clientside HTML/CSS/Javascript and the serverside node/database. By backend, I am referring to the data processing off a job queue like Kue/Gearman

Question: How would you structure such a site?

Meteor-backed server (or node instance) for the frontend, and an Express server with Kue/Redis on the backend? Or 2 seperate Meteor servers, one for the frontend one for the backend? Or 1 single Meteor server for both serving the frontend and doing the backend processing?

And what are your justifications for your recommendation? Thank you! :)

Nyxynyx
  • 61,411
  • 155
  • 482
  • 830

1 Answers1

2

By backend, I am referring to the data processing off a job queue like Kue/Gearman

As this sounds like a rules/processing engine uncoupled from the "front end" that delivers both the "clientside HTML/CSS/Javascript" and the "serverside node/database" what I think may well serve your needs is a DDP client that can subscribe to the meteor server-side publications and queue up the jobs accordingly (using an engine such as Kue).

Such a client could live completely independent of the Meteor app in its own environment. This way, you can still leverage all the reactivity goodness you seek in Meteor, yet use more established processing tools for queue-based jobs that run independently of the UI. With the DDP client, you can also hook back into the UI to inform clients when jobs complete by leveraging the subscriptions.

Here is a node DDP client that may prove helpful. https://github.com/oortcloud/node-ddp-client

Hope this helps!

Community
  • 1
  • 1
TimDog
  • 8,758
  • 5
  • 41
  • 50
  • Thanks for the response. I'm a little confused here: Why do you have the "backend" that runs Kue and processes the jobs, also run a DDP client to subscribe to the "frontend" meteor publications? Shouldnt the "frontend" app be notified of changes (like job failure/completion) by the "backend"? *(I reread your post and figured this is done by the hookback)* – Nyxynyx Dec 30 '12 at 05:29
  • The way I think of it is that the user uses the "frontend" app to define a job, which is passed to the "backend" server via a `POST` request. The "backend" receives this `POST` request, adds it to the job queue, procsses the job when its time, and marks it as complete. The completion of a job should trigger a "reactive" update to the user who may still be on a page on the "frontend" app. Should the "frontend" app access the Redis/Kue database directly if it wants to check the status of all jobs belonging to a user? Or access them indirectly through the "Backend"? – Nyxynyx Dec 30 '12 at 05:30
  • Good questions. To clarify: you can store all of your data in a mongo collection in the Meteor environment (the "frontend" in our discussion); then your "backend" -- with its own DDP client -- is simply the jobs engine that uses the "frontend"-backed collections to orchestrate its processing through DDP -- both in subscribing to collection changes and remotely invoking [Meteor.methods](http://docs.meteor.com/#meteor_methods) for "reactive" updates to the UI. The "frontend" follows Meteor conventions and should be oblivious to the "backend" processor, although both share the same mongo db. – TimDog Dec 30 '12 at 16:14
  • 1
    `DDP` replaces conventional communication between the two environments altogether...the "frontend" should never `POST` to the "backend" at all. The "frontend" hums along merrily without knowing how the jobs are being processed; the "backend" only changes the mongodb collections via DDP, and the UI reactivity orchestration is handled entirely by the meteor framework once the collection has changed externally through the DDP client. – TimDog Dec 30 '12 at 16:21