0

I have a distributed monitoring system that collects and gathers monitoring data like CPU utilization, database performance metrics, network performance into a backend store. Other applications need to consume these data like real-time calculating(for a resource scheduler) , for system monitoring(to system administrator using monitoring dashboard), for historical analytic(to operation and analyzer program to modeling the resource using pattern for future capacity planning and business system activity analysis).

The dataset size is about 1.2 billion entries in the data store for 9 months. (all in OpenTSDB like format)

Previously I used an Elasticsearch cluster as the backend data store solution and decide to find a better one.

I am looking at Couchbase or VoltDB cluster but still in investigation stage so need some input from here who has the similar experience.

Major questions are as below:

  1. Which backend store solution is good for my scenario? (Couchbase or VoltDB)?

  2. I have to rewrite my data aggregator code (which is in golang). Couchbase provide a good golang SDK client but VoltDB's go driver is only in community level with limited function. So are there any better implementation to communicate with voltdb in golang?

  3. Any suggestion or best practice on it?

DaveR
  • 9,540
  • 3
  • 39
  • 58
nntp
  • 43
  • 1
  • 5
  • Can you give some more information on how often the consuming of the data will be? Couchbase can definitely keep up with the writes in this scenario. For reads it will depend on the data modeling and which of the 3 means of accessing data you need and what performance characteristics you need out of this DB. – NoSQLKnowHow Sep 04 '15 at 03:09
  • Thanks Kirk. The interval that gathering data from target node and services running on it is about 30s(min) to 5min(max). Our monitor agent 's monitoring interval is configurable during the running time. – nntp Sep 07 '15 at 07:36
  • We leverage OpenTSDB's data model and appended some our own specifics metrics into it. (let's say A customized OpenTSDB model). Couchbase's View feature is definitely the one that attracted me so that i can create different view upon the gathered data and analysis them in different way. All i concern is adopt Couchbase cluster as the monitoring system backend is the right way or not. Do you know any couchbase use cases that really happened in this way? Thanks. – nntp Sep 07 '15 at 07:41
  • There is no "right" way in tech. Anyhow, yes there are definitely people that do things like this with Couchbase. I looked at the OpenTSDB model and it is not JSON, but that should be easy enough to put into JSON to go into Couchbase and use views. One thing to remember with Couchbase views is that they are eventually consistent and more of a map reduce view. Sizing of your cluster will be critical. Depending on the nature of your querying it might even be worthwhile to take a look at the new Couchbase 4.0 query capabilities. Also, read this. http://bit.ly/1Hx3a74 – NoSQLKnowHow Sep 08 '15 at 04:00

2 Answers2

1

There isn't too much in the way of usage patterns here, but it sounds like the kind of app people use VoltDB for.

As for the Golang client, we'd love some feedback as to how to make it more suitable if it's specifically missing something you need. You can also use the HTTP/JSON query interface from any language, including Golang. More info on that here: http://docs.voltdb.com/UsingVoltDB/ProgLangJson.php

John Hugg
  • 380
  • 1
  • 2
-1

If you would like to leverage your existing model, take a look at Axibase Time-Series Database. It supports both tcollector network and http protocols. Rule engine and visualization are built-in.

The fact that ATSD is based on HBase may be an asset or a liability depending on your prior experience with it :)

URL to tcollector integation: http://axibase.com/products/axibase-time-series-database/writing-data/tcollector/

Disclosure: I work for the company developing ATSD.

kghamilton
  • 23
  • 7
Sergei Rodionov
  • 4,079
  • 6
  • 27
  • 44