8

I am using hortonworks sandbox.
creating topic: 

./kafka-topics.sh --create --zookeeper 10.25.3.207:2181 --replication-factor 1 --partitions 1 --topic lognew

tailing the apache access log directory: 

tail -f  /var/log/httpd/access_log |./kafka-console-producer.sh --broker-list 10.25.3.207:6667 --topic lognew

At another terminal (of kafka bin) start consumer: 

./kafka-console-consumer.sh --zookeeper 10.25.3.207:2181 --topic lognew --from-beginning

The apache access logs are sent to the kafka topic "lognew".

I need to store them to HDFS.
Any ideas or suggestions regarding how to do this.

Thanks in advance.
Deepthy

prabhugs
  • 742
  • 7
  • 20
Deepthy
  • 139
  • 5
  • 14

2 Answers2

6

we use camus.

Camus is a simple MapReduce job developed by LinkedIn to load data from Kafka into HDFS. It is capable of incrementally copying data from Kafka into HDFS such that every run of the MapReduce job picks up where the previous run left off. At LinkedIn, Camus is used to load billions of messages per day from Kafka into HDFS.

But it looks like it's replaced with gobblin

Gobblin is a universal data ingestion framework for extracting, transforming, and loading large volume of data from a variety of data sources, e.g., databases, rest APIs, FTP/SFTP servers, filers, etc., onto Hadoop. Gobblin handles the common routine tasks required for all data ingestion ETLs, including job/task scheduling, task partitioning, error handling, state management, data quality checking, data publishing, etc. Gobblin ingests data from different data sources in the same execution framework, and manages metadata of different sources all in one place. This, combined with other features such as auto scalability, fault tolerance, data quality assurance, extensibility, and the ability of handling data model evolution, makes Gobblin an easy-to-use, self-serving, and efficient data ingestion framework.

Anatoly Deyneka
  • 1,238
  • 8
  • 13
  • Thank you for you reply ... **I tried to run camus but getting the following error:** com.linkedin.camus.coders.MessageDecoderException: java.lang.InstantiationException: com.linkedin.camus.example.schemaregistry.DummySchemaRegistry – Deepthy Nov 23 '15 at 08:30
  • I guess you need to properly setup the project: http://etl.svbtle.com/setting-up-camus-linkedins-kafka-to-hdfs-pipeline – Anatoly Deyneka Nov 23 '15 at 09:39
  • Other than using camus and gobblin is there any other way to load kafka topic to hdfs? – Deepthy Nov 24 '15 at 05:28
  • We have stopped on camus because it's successfully used in linkedin. I guess you can find other ways but camus and goblin are most popular for this moment. – Anatoly Deyneka Nov 25 '15 at 06:34
2

You have several other options as well:

  • Use Apache Flume to read messages from Kafka and write them to your HDFS. There are several examples of how you can set it up, but one article from Cloudera covers that topic quite well. They even named the solution Flafka ;)
  • Use Kafka HDFS Connector, which is quite simple to set up. However, it would require Confluent Kafka (which still is open sourced).

We tested both quite successfully.

wjp
  • 148
  • 9
  • 1
    If possible add content from the links to your answer. This way, if the links go dead, your answer is still useful – mnwsmit Jun 14 '16 at 14:06
  • 1
    They're quite big articles, but I will try to explain some of the concepts. – wjp Jun 15 '16 at 07:51
  • How do you compile kafka hdfs connector? It says it needs common and schema repository master which I downloaded but I am unable to build using instructions at https://github.com/confluentinc/common. Filed a bug: https://github.com/confluentinc/common/issues/37 – Adrian Sep 01 '16 at 18:11