-1

broker/docker-compose.yml

---
version: '2'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.0.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-enterprise-kafka:6.0.1
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

ksql/docker-compose.yml

---
version: '2'

services:
  ksqldb-server:
    image: confluentinc/ksqldb-server:0.14.0
    volumes:
      - ./my_quiries/select.sql:/opt/my_quiries/select.sql
    hostname: ksqldb-server
    container_name: ksqldb-server
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: localhost:29092
      KSQL_KSQL_SERVICE_ID: demo_app
      KSQL_KSQL_QUERIES_FILE: /opt/my_quiries/select.sql
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"

I want to run Ksqldb (StandAlone) in distributed fashion. so I tried to run broker and ksqldb from individual compose file from different linux machine one is wsl2 and another is linux vm. but I am stuck with this error to run ksqldb compose file

[2021-01-06 03:17:51,522] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:780)

when my broker is running successfully on port localhost:29092.

Saiful Islam
  • 186
  • 1
  • 3
  • 13

2 Answers2

1

Using localhost in ksql is not accessing the host (it is just the localhost of the container itself). You need to provide the docker host IP (when on linux) or you can use host.docker.internal (when on windows).

Another option is to just use docker networking but using two independent docker-compose files will produce two networks (broker_default And ksql_default if I‘m not wrong). So ksql isn‘t able to connect to your broker as it is isolated in it‘s own network. You can solve this with f.e. following:

  • Create some shared network like docker network create shared_network
  • In your docker-compose files add this network to ksqldb and kafka to be able to communicate

ksql:

version: '2'

services:
  ksqldb-server:
    ...
    networks:
    - shared_network

networks:
  shared_network:
    external: true

broker:

services:
  zookeeper:
    ...
    networks:
    - internal

  broker:
    ...
    networks:
    - internal
    - shared_network

networks:
  internal:
  shared_network:
    external: true
x4k3p
  • 1,598
  • 2
  • 22
  • 42
  • Your solution working fine when both compose file in same machine & initially it was my problem. but if both are in different machine not working. if any solution for different machine case.. it will great help for me. – Saiful Islam Jan 08 '21 at 04:14
  • As OneCricketeer already mentioned in the comments, just replace the localhost part with the IP of the host which is accessible by the other host. And if you like to run your setup on different machines you should include such information in your question as it is unclear otherwise. – x4k3p Jan 08 '21 at 04:28
0

I want to run Ksqldb (StandAlone) in distributed fashion.

That's not a valid reason for using individual Compose files. If you were to scale a service, you'd either use multiple copies of different service blocks in one file, or Swarm/Kubernetes to scale the running replica count of the service.

As the other answer indicates, KSQL_BOOTSTRAP_SERVERS: localhost:29092 should rather be KSQL_BOOTSTRAP_SERVERS: broker:9092 where the services are on the same Docker network (which they are in the example Compose file I suspect you copied from)

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • Actual reason of trying like this is on my deployment environment broker and ksqldb may be in different machine thats why i divided the compose file to check how to connect when they are in individual compose file. And I tried with KSQL_BOOTSTRAP_SERVERS: broker:9092. But it is not working. – Saiful Islam Jan 07 '21 at 03:46
  • You need to use a shared bridge/overlay network in the event that you are using separate files and especially true if you're using separate machines. Like answered, though, if using separate machines that's were you'd use Docker Swarm or Kubernetes – OneCricketeer Jan 07 '21 at 04:47
  • `that's were you'd use Docker Swarm or Kubernetes` I am using docker desktop only. would you please explain is it possible or not in my environment without Docker Swarm or Kubernetes? – Saiful Islam Jan 07 '21 at 06:18
  • Docker Desktop supports both those options – OneCricketeer Jan 07 '21 at 06:23
  • If you do not want to use those, then you must 1) add your external host IP to the advertised listeners 2) expose the listener port via the ports Compose config 3) use that hosts address on the external machine(s) – OneCricketeer Jan 07 '21 at 06:25
  • It will great help if you share any example or link to follow. – Saiful Islam Jan 07 '21 at 07:27
  • 1
    I don't have one. This is purely a networking issue specific to your setup. The best I can do is this https://www.confluent.io/blog/kafka-listeners-explained/ but replace each instance of localhost with your docker host"s external, LAN IP – OneCricketeer Jan 07 '21 at 07:31