0

I am very new with Kafka and Streaming Data in general. What I am trying to do is to ingest data which is to be sent via http to kafka. My research has brought me to the confluent REST proxy but I can't get it to work.

What I currently have is kafka running with a single node and single broker with kafkamanager in docker containers.

Unfortunately I can't run the full confluent platform with docker since I don't have enough memory available on my machine.

In essence my question is: How to setup a development environment where data is ingested by kafka through http?

Any help is highly appreciated!

LukasM
  • 43
  • 8
  • 1
    Hi and welcome to SO, please rewrite your question to match the Q&A Format, I don't know much to answer but "Buy more Memory" otherways. Make sure to read [ask] – LuckyLikey Jan 14 '19 at 15:17

2 Answers2

1

You don't need the "full Confluent Platform" (KSQL, Control Center, included)

Zookeeper, Kafka, the REST proxy, and optionally the Schema Registry, should all only take up-to 4 GB of RAM total. If you don't even have that, then you'll need to go buy more RAM.

Note that Zookeeper and Kafka do not need to be running on the same machines as the Schema Registry or REST proxy, so if you have multiple machines, then you can save some resources that way as well.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
1

To run one Kafka broker, zookeeper and schema registry, 1Gb is usually enough (in dev).

If you do not want for some reason to use Confluent REST proxy, you can write your own. It's quite straightforward: "on request, parse your incoming JSON, validate data, construct your message (in Avro?) and produce it to Kafka".

In this article, you'll find some configuration to press Kafka and ZK on heap memory: https://medium.com/@saabeilin/kafka-hands-on-part-i-development-environment-fc1b70955152

Here you can read how to produce/consume messages with Python: https://medium.com/@saabeilin/kafka-hands-on-part-ii-producing-and-consuming-messages-in-python-44d5416f582e

Hope these help!

saabeilin
  • 610
  • 7
  • 15