3

I have to analyze Gzip compressed log files which are stored on a production server using Hadoop related tools .

I can't decide on how to do that, and what to use, here are some of the methods i thought about using (Feel free to recommend something else):

  • Flume
  • Kafka
  • Map reduce

Before i could do anything, i need to get the compressed files from the production server and process them then push them into Apache HBase

Mohamed Benkedadra
  • 1,964
  • 3
  • 21
  • 48
Yaswanth
  • 483
  • 1
  • 9
  • 25
  • what about use apache drill? – SQL.injection Nov 24 '15 at 11:11
  • How big are the logs (uncompressed)? – Mark Waterman Nov 25 '15 at 00:33
  • Mark, the logs are actally 200MB each and will be mostly 10 files. – Yaswanth Nov 25 '15 at 10:29
  • 1
    So you're talking about maybe 2GB of log data per day? Is hbase a firm requirement, because it seems like overkill for so little data (you can fit several years' worth on a single HDD). If you have some flexibility, @MarkoBonaci's Databricks suggestion is spot on, or look at something like [Splunk](http://www.splunk.com/), which was pretty much designed for this scenario. – Mark Waterman Nov 25 '15 at 19:30
  • Thanks Mark, but still we haven't confirmed about that. – Yaswanth Nov 26 '15 at 02:37

4 Answers4

5

Depending on the size of your logs (assuming that the computation won't fit on a single machine, i.e. requires a "big data" product), I think it might be most appropriate to go with Apache Spark. Given that you don't know much about the ecosystem it might be best to go with Databricks Cloud, which will give you a straightforward way of reading your logs from HDFS and analyzing using Spark transformations in a visual way (with a Notebook).

You can find this video on the link above.
There's a free trial so you can see how that would go and then decide.

PS I'm in no way affiliated with Databricks. Just think they have a great product, that's all :)

Marko Bonaci
  • 5,622
  • 2
  • 34
  • 55
1

You have mixed many inter-related concepts which are not alternatives to each other.

Have a look at hadoop ecosystem

Apache Map Reduce is : A YARN (Yet Another Resource Negotiator) based system for parallel processing of large data sets. It provides simple programming API.

Apache Kafka is a Distributed publish-subscribe system for processing large amounts of streaming data. You can treat Kafka as a simple "Message Store"

Apache Flume is specially designed for collection, aggregation, and movement of large amounts of log data (in unstructured format) into HDFS system. It collects data from various HTTP sources and web servers.

Once data is imported from Flume to HDFS, it can be converted into structured data with PIG or Hive and reports can be generated in Structured form. PIG or HIVE runs a series of Map Reduce Jobs to process this data and generate reports.

Have a look at this article to have better understanding on log file processing architecture.

Ravindra babu
  • 37,698
  • 11
  • 250
  • 211
  • Sorry Ravindra, I have not formatted my question properly before. Just now I have edited it again, could you please check it once again. – Yaswanth Nov 24 '15 at 16:05
0

Each of the tools you mentions is doing something else -

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data

Apache Kafka is publish-subscribe messaging rethought as a distributed commit log Map reduce is more of design pattern for processing data.

My suggestion is to define better what you really look for an examine the relevant tools.

Tom Ron
  • 5,906
  • 3
  • 22
  • 38
0

As I have log files stored in production server, I am going to copy these files into HDFS and I have written mapreduce program to process it.

I think @Marko Bonaci answer is valid, we can try with spark to analyze the log files.

Thanks all for your valuable input.

Yaswanth
  • 483
  • 1
  • 9
  • 25