-3

Requirement : Sending data from an android app or iOS app to big data ecosystem using Kafka and store it for real time analysis or near real time.

1- How to send data from Android App to Kafka (Is rest API only options)? Or we can connect to the kafka with App directly.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
D-2020365
  • 82
  • 1
  • 11
  • Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. – Sergey Kovalev May 18 '18 at 10:44
  • @SergeyKovalev: Thanks for suggestion, I make the question to one only for now. I found some place people asked multiple question so i did. – D-2020365 May 18 '18 at 11:03
  • Keep in mind - mobile devices have very unreliable connections and are suspect to dropping packets or duplicate data while switching networks. You can try it, but it would likely lower battery life handling long running connections. It wouldn't be much different than how anyone would be collecting MQ messages from mobile devices, if they are – OneCricketeer May 18 '18 at 12:48
  • @cricket_007: Scenario- User opened an app and did some action on that App like post a message. In that case, I want to do some real-time analysis of data that user post. will it be better to store the data in RDBMS and do that Analysis?. I thought like that take the daily migration to hive for more analysis. I am looking something in big data ecosystem. – D-2020365 May 20 '18 at 09:24
  • Okay... And what do you have against using HTTP to some service which forwards data to Kafka? My point here - if mobile devices can send data to Kafka, how are you protecting any other random person from also producing (or consuming) that data by connecting to your external Kafka port? – OneCricketeer May 20 '18 at 14:55
  • @cricket_007: I am not against to any HTTP request but looking for some other options too. – D-2020365 May 22 '18 at 11:19
  • REST is not the only way. You'll be able to find several posts online about MQTT, AQMP, or maybe JMS (at least for Android). The problem will be, though, you still need to connect *those services* to Kafka. For example https://www.landoop.com/blog/2017/01/mqtt-kafka-connect-with-ais-data/ But I would argue exposing a one way channel for purely sending data is better than exposing your entire Kafka cluster – OneCricketeer May 22 '18 at 13:42
  • Yes, I agree with your point and will not expose the Kafka cluster. i will create REST service that will connect to cluster. if load increase will look some other technology like load balancing to support rest services. – D-2020365 May 22 '18 at 14:28

1 Answers1

2

There is a REST Proxy for Kafka, so you can send messages over HTTP from your mobile app directly to Kafka, and from there process them further in Kafka, route them to Hadoop, etc etc.

Disclaimer: I work for Confluent, who lead the development of the open-source REST proxy

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92