3

I am trying to run Kafka in cluster mode using two instances of schema registry but I am not quite sure how to configure the second instance so that it takes over in case the first one is down.

Here's the properties file for the first schema-registry instance: 

port=8081

# The address the socket server listens on.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=http://0.0.0.0:8081

# Zookeeper connection string for the Zookeeper cluster used by your Kafka cluster
# (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
kafkastore.connection.url=localhost:2181,localhost:2182,localhost:2183

# Alternatively, Schema Registry can now operate without Zookeeper, handling all coordination via
# Kafka brokers. Use this setting to specify the bootstrap servers for your Kafka cluster and it
# will be used both for selecting the master schema registry instance and for storing the data for
# registered schemas.
# (Note that you cannot mix the two modes; use this mode only on new deployments or by shutting down
# all instances, switching to the new configuration, and then starting the schema registry
# instances again.)
#kafkastore.bootstrap.servers=localhost:9092

# The name of the topic to store schemas in
kafkastore.topic=_schemas

# If true, API requests that fail will include extra debugging information, including stack traces
debug=false

How should the second file look like so that they can both communicate with zookeeper and achieve high availability?

Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156

1 Answers1

2

You can use either the Kafka leader election or Zookeeper leader election.

The only thing you need to change between two instances on the same machine connected to the same Kafka/Zookeeper is the port and listeners property

To appropriately configure high availability, you need a HTTP load balancer for giving one address for all instances.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Assuming that I have two instances of Schema Registry up and running, one listening on 8081, and another listening on 18081; Do both of them accept `GET` requests? Although I am able to get a response when calling `kafka-host:8081/subjects`, I am not able to get a response for `kafka-host:18081/subjects`. Is this a normal behaviour? – Giorgos Myrianthous Dec 11 '18 at 08:03
  • Only one master can be eligible at a time, but I would think both can accept requests – OneCricketeer Dec 11 '18 at 18:15
  • I am not able to get any response when calling the second instance. Is this normal? Same goes for Kafka REST proxy instances. – Giorgos Myrianthous Dec 12 '18 at 08:28
  • I'm not entirely sure why you'd need two on one machine... I assume you're just running the start script twice? I could try it – OneCricketeer Dec 12 '18 at 15:42
  • I am running two instances on the same machine for testing purposes. On production environments these two instances will be running on different hosts. – Giorgos Myrianthous Dec 12 '18 at 15:50
  • Generally, I'm running things in Docker. Not sure if two Docker processes or not makes a difference... – OneCricketeer Dec 12 '18 at 15:55