Looks like AWS Kinesis Firehose is a way to ingest data (buffered together) to an object, is there a Kafka counterpart for this kind of functionality?
Asked
Active
Viewed 2,629 times
1
-
This was supposed to be done by Apache Flume. – oakad Feb 19 '19 at 04:18
1 Answers
4
Yes, Kafka Connect, which is part of Apache Kafka.
You can use it with a variety of connectors, for streaming data in from a source (RDBMS, JMS, MQTT, CSV, etc etc) to a Kafka topic, and from a Kafka topic to a target (RDBMS, S3, BigQuery, HDFS, etc etc)
Kafka Connect in action:
- https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/
- https://www.confluent.io/blog/blogthe-simplest-useful-kafka-connect-data-pipeline-in-the-world-or-thereabouts-part-2/
- https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-3/
Disclaimer: I work for Confluent.
Robin Moffatt
- 30,382
- 3
- 65
- 92
-
1Is it possible to connect data from Kinesis Firehose (or its S3 products) to Kafka ? – Oded Rosenberg Nov 22 '20 at 17:33