2

I wasn't entirely sure how to word my question in the title so apologies if it's confusing.

I'd like to build a system that would function as a sort of information dashboard for my home. It would consist of a number of hardware and software components that would ultimately result in a simple, clean website with real-time displays of a number of analog sensors such as temperature, wind speed and direction, etc.

I've got a good idea of what I'm going to do for the hardware, as well as for displaying the information; my question has to do with the communication between the hardware and web server.

I'd like the hardware to fire messages at a fairly fast rate so I don't think HTTP POST will suffice. I'm also not extremely concerned with receiving 100% of the messages but receiving as many as possible is definitely a plus. The data will be coming from the hardware, populating some sort of database (likely Redis).

So far, I've researched a couple of things but I'm not sure I'm heading in the right direction. I've looked in to message-oriented middleware such as RabbitMQ but I'm not convinced I need the overhead. I've also looked into Redis Pub/Sub which seems like a more appropriate solution since I'd like the web app to chart out say the last 5 minutes of data but even then I'm not certain. Can I just fire UDP packets to a custom-built listener?

I'm pretty certain the hardware will be two stages (a uC feeding a small embedded linux machine) so you might even liken this to desktop software firing messages to a web server as quickly as possible.

I'm venturing into an area that I know absolutely nothing about so any guidance is much appreciated.

Jean-Charles
  • 366
  • 2
  • 11
  • I was going to edit your tags to add 'realtime-data data-collection' and anything else having to do with realtime data collection, but that would exceed the recommended tag limit. You might consider removing redis, message-queue and rabbitmg, as your question really appears to be: "how can I collect data from realtime hardware devices in a way that easily integrates with nodejs?" – Rob Raisch Jun 01 '11 at 02:58
  • I updated the tags per your advice but I'm not sure it's the nodejs integration I'm concerned with; nodejs will be used for presenting the data, not necessarily in the data collection process (though that's the stage that I am most uncertain of, so it could). – Jean-Charles Jun 01 '11 at 03:21

3 Answers3

1

To communicate between your data acquirers and collectors, you might consider the industry standard Modbus TCP protocol. (In a previous life, I wrote network code for programmable controllers.)

I'm sure there are libraries available for most microcontrollers, though they might not be open-source, but I doubt a JS version of Modbus exists so you'd need to write the server-side lib yourself. As I recall Modbus isn't particularly complex, especially if you don't use some its more esoteric features. Of course, writing this got me thinking how I'd write such a thing and lo and behold, it's already been done for nodejs! (One of the many reasons I love the nodejs community!)

So thats the easy answer...now, with my hacker-hat firmly in place...

You mention that your HW will feed one or more "small embedded Linux machine(s)".

Have you considered running nodejs on each data collector? If the size of nodejs' executable is the issue, I'm sure there are large parts of its out-of-the-box functionality that could be removed or moved into modules.

I realize what I'm recommending is not a small undertaking--porting an application the size and complexity of nodejs/V8 to a new platform is certainly challenging--but I strongly suspect nodejs' event-driven design would be an excellent match for data acquisition, discrete manufacturing, process control, and other manufacturing applications.

Rob Raisch
  • 17,040
  • 4
  • 48
  • 58
  • Fortunately (or unfortunately, depending on how you look at it), I'll only be using a single uC with only a handful of sensors (few enough to just use the pins on that uC), so I'll be communicating over a single, direct serial connection. Thanks for the tip; perhaps for the second iteration. – Jean-Charles Jun 02 '11 at 02:16
0

What's wrong with http post? If you are using node.js as your web server it should be plenty fast enough. You are already coding the presentation layer in node so that's one less tool you will have to use, and in this case it's a perfect fit. Keep it simple and stick with node.

Lyn Headley
  • 11,368
  • 3
  • 33
  • 35
  • I guess there is nothing wrong but the engineer in me just felt like the round-trip over HTTP was a lot of overhead for what will be a little bit of data fired frequently. – Jean-Charles Jun 01 '11 at 15:38
0

Like the other poster mentioned, your're not going to have issues with http post. Node's http implementation is built for high concurrency.

Personally, I think I'd go with:

  1. Hardware device output ->
  2. Linux box fires an http post directly to your central server (node.js) ->
  3. Take your changes and immediately publish them to your web client via socket.io (real time transport for the browser). https://github.com/LearnBoost/Socket.IO/

Socket.io is probably the best out of the box real time transport for node.js <==> browser

If you want persistence, redis isn't bad (plus you get the free pub/sub) if your data fits into that model, which it probably does.

There's no reason you couldn't use a reglar tcp based connection also (net module) if thats your bag. Unless you want the data to be unreliable, I wouldn't go to udp. Thats more lossy streaming media oriented.

I really doubt that you're going to have enough messages to worry about a dedicated message queue. Rabbitmq introduces features like queue persistence and is built for incredibly high throughput. Probably orders of magnitude more than what you're after.

You might look at a library like zeromq for which there are node bindings: https://github.com/JustinTulloss/zeromq.node. Its like a topic/other type of exchange in rabbitmq, but without a central message queue node (one calls this brokerless). That way you can just talk directly to your linux/hardware nodes, but still get the message queue like interface -- you publish your hardware updates on a 'channel' and your server nodes listens for such updates.

Josh
  • 12,602
  • 2
  • 41
  • 47
  • So since I want persistence and graphs over set durations, which model would you go with? Low level hardware (uC) -> high level hardware (embedded linux device) -> redis (pub/sub) <- nodejs (presentation) **or** Low level hardware (uC) -> high level hardware (embedded linux device) -> POST to nodejs which is immediately reflected in nodejs (presentation) and sent to redis (to archive)? Is the first model even possible (pub/sub and persistent storage)? – Jean-Charles Jun 01 '11 at 15:37
  • To avoid complexity, I think I'd use the latter approach. Push data from embedded linux device directly to node (using whatever transport you prefer), node handles dispatching new data to connected clients, dispatching data to redis (or whatever) for persistence and recall, and serving graphs/presentation to newly connected clients (using redis to recall data stored over time). – Josh Jun 01 '11 at 19:00
  • Under the covers HTML POST and redis PUBSUB are pretty much the same; they are both ASCII character-based, line-oriented protocols over TCP so any concerns related to using POST would be the same for redis. The only concern I would have using transactional line-oriented protocols like HTTP and PUBSUB would be the *enormous* cost in setting up a new TCP session for each request. Using a packet or stream-oriented protocol over long-lived TCP connections would seem to make better sense. – Rob Raisch Jun 01 '11 at 19:14
  • how much more overhead would it realistically add given http keepalive? its probably significantly more fragile, i guess. but over a local network, i would imagine packet size isn't a terrible issue, and running things through a http parser is not exactly the end of the world – Josh Jun 01 '11 at 19:34