0

I need to analyze data stored in multiple databases. Each of them contains a few TB of logs and sensor data.

If I use Splunk or ElasticSearch/Kibana, I see 2 solutions:

  1. Batch import everything

  2. Write some scripts to import only the data I need, on demand

This is not optimal because with 1. I duplicate TB of data, and with 2. I need some custom logic that won't be integrated with my analytics tool.

My questions are:

  • Is there a way to perform analytics in place with Splunk or Kibana, i.e. skip the importing step and just read everything from the existing databases?

  • Or is there a tool that would automatically fetch the relevant data and only the relevant data?

Raffi Khatchadourian
  • 101
  • 3
MasterScrat
  • 314
  • 3
  • 9

1 Answers1

0

To be able to search you data quickly i advice Elasticsearch Depending on what the logic behind relevant data should be you can use ElasticSearch Rivers http://www.elasticsearch.org/guide/en/elasticsearch/rivers/current/

This allows Elastic search to pull all (relevant) data. My experience says the TB of data you have stored will be reduced to several GB in ElasticSearch

Brian van Rooijen
  • 197
  • 4