-1

Which would be the best way to save timeseries data to an external file system for further processing in another platform?

By best I mean:

  • Less possible coding
  • As much endurance to heavy load as possible
  • Integrated with TB (or using TB tools or workarounds inside TB)

The solution could be near realtime, or batch, as the final destination is for analysis and does not require last minute information.

I thought of a few ways, but I'd like to get some advice or thoughts on which one would be more convenient. And of course, I will appreciate any other ideas I haven't thought of.

For the near realtime mode:

  1. In the rule chain, save the data with a node following the Save Timeseries node.

    a. Could this affect the performance significantly? as it will add a save operation to file system for each call to the node.

    b. Is there any code example of such a node?

  2. Use Rest API Call node to POST timeseries data to an endpoint and generate the file there at the endpoint.

    a. Does this have any chance of being performant?

    b. Is there any code example of how to do it?

  3. Use Kafka node to forward the timeseries data to a Kafka server and from there to its final destination.

    a. This introduces another tier (Kafka) which requires another kind of expertise and more resources.

    b. Has anybody worked with this node successfully? And, would you mind sharing and example?

For the batch mode:

  1. Find out which tables are used by TB inside Cassandra, and code a script to extract information directly from the database.

    a. This would require to have a good knowledge of the TB data model inside Cassandra and of Cassandra itself to be able to write the scripts. So it doesn't look like a very natural/integrated way of solving the problem.

  2. Create a rule chain that triggers a query at certain time intervals, to retrieve a time period of TB saved timeseries data, and then uses some of the near realtime options to save the file in one operation.

Ric
  • 65
  • 9

1 Answers1

2

I will try to answer as best as I can

If you are working with community TB and have access to the codebase, i would write a "file" node that can be attached post "save timeseries" and write to the file system. You could even write an "ftp" node for that matter if applicable. Documentation for writing a custom rule node is available in the TB documentation.

If you are strictly a user of the TB application and have access to rule chain, then i would use a MQTT node to send data out to an external system and then process the data from there. A "low-code" option can be Node-RED where you can set up a MQTT broker, receive data and use the "file" node to write to the file system.

In regards to saving it as a CSV or JSON, the timeseries data is in JSON by default, so you can pass that along. If it is CSV, you could again either write a CSV node or do the transformation on the receiving end before writing to the file system.

Hope this helps.

RubikX
  • 38
  • 5
  • @everybody else that commented on this post. My question was edited by another member, and he corrected some words that I agree had mistakes, but also trimmed some complete sentences that were mainly saying "thank you to the ones that takes the time to answer my question". I'm relatively new to the site and I would like to understand something: it is not customary to thank other people's time or attention here? Because it feels really strange for me. However, if this is the way, I'll stick to it, but I would appreciate clarification on the matter. Thank you! – Ric Dec 28 '21 at 14:53
  • @Ric - I appreciate your intent to be polite and thankful. Because Stack Overflow is built to be a definitive repository of programming information, the community tends to edit questions and answers down to focus on just the meat of the post. It's a different aspect of the culture here that takes a bit of getting used to. For more, I recommend reading ["Should 'Hi', 'thanks', taglines, and salutations be removed from posts?"](https://meta.stackexchange.com/questions/2950/should-hi-thanks-taglines-and-salutations-be-removed-from-posts). – Brad Larson Dec 28 '21 at 16:46
  • @BradLarson - 100% understood, and thank you very much for the information!!! – Ric Dec 28 '21 at 20:54