11

I'm investigating the merits of using BigQuery for gaining insights into application logfiles. The logs are produced by Java and C# applications, most of them on cloud-based VMs. I'm interested to hear whether others have done this and of the relative merits of BigQuery vs ElasticSearch/Logstash/Kibana.

The advantage of BigQuery seems to be that it can deal with huge amounts of data whereas the ELK solutions seems maybe better suited to the non-structured nature of logfiles, especially when they come from different systems.

I'd also like to display information on a dashboard. Kibana seems to be very good for that. How easy is it to create dashboards using the Google solution (using google sheets, etc)?

Thoughts, use-cases?

user3628387
  • 139
  • 1
  • 2
  • 9

1 Answers1

11

2017 update: Elastic officially supported on GCP

Elasticsearch and BigQuery work great together. BigQuery will take as much data as you have and query it in any way you want in seconds. Meanwhile a well tuned Elasticsearch installation will give you answers in less than a second, but only for certain queries over a limited amount of data.

See this post by Ory at Rounds, where they detail how they use both:

https://medium.com/@oryband/collecting-user-data-and-usage-ffa84c4dba34

The two top titles that summarize their reasons to do both:

  • Live Data with Elasticsearch
  • Big Data with Google BigQuery
Michael Laffargue
  • 10,116
  • 6
  • 42
  • 76
Felipe Hoffa
  • 54,922
  • 16
  • 151
  • 325
  • http://www.rounds.com/blog/collecting-user-data-and-usage/ since https is not openning – Kassem Shehady Feb 13 '17 at 16:43
  • But how you save logs to elastic search? do you use logstash? – Kassem Shehady Feb 13 '17 at 16:44
  • 2
    @KassemShehady I use filebeat to ship the logs from the clients, and logstash on the servers to process them before sending them off to elasticsearch – user3628387 Aug 18 '17 at 12:35
  • 1
    I finally uses apache beam ElasticSearchIO https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java – Kassem Shehady Aug 19 '17 at 18:32
  • 1
    new link for the article: http://rounds.com/www.rounds.com/blog/collecting-user-data-and-usage/index.html – Zhe Li Oct 24 '17 at 13:47
  • 2
    a functioning link to that article >< https://medium.com/@oryband/collecting-user-data-and-usage-ffa84c4dba34 – ptim Mar 03 '20 at 03:56