8

I don't know how to configure Kafka for Symfony messenger. Everything works for rabbitmq (I created messenger and messenger handler):

.env:

MESSENGER_TRANSPORT_DSN=amqp://user:password@myhost:5672/%2f/messages

config/packages/messenger.yaml

framework:
  messenger:
    transports:
      async: "%env(MESSENGER_TRANSPORT_DSN)%"

.env

MESSENGER_TRANSPORT_DSN=enqueue://node-1.kafka.myhost.com:9092/%2f/messages

config/packages/messenger.yaml

framework:
  messenger:
    transports:
      async: "%env(MESSENGER_TRANSPORT_DSN)%"

Please, give me the best example. Thanks!

Mike Doe
  • 16,349
  • 11
  • 65
  • 88
Arkadiusz G.
  • 1,024
  • 10
  • 24

1 Answers1

15

My develompent: Docker + Centos 7 + PHP73, NGINX.

Solution for this configuration:

1. Install php-rdkafka (important: version 3.1.x!, change path to php ;))

yum -y install make librdkafka-devel && git clone --branch 3.1.x https://github.com/arnaud-lb/php-rdkafka.git && cd php-rdkafka && /path/to/php73/root/bin/phpize && ./configure --with-php-config=/path/to/php73/root/bin/php-config && make all -j 5 && make install

2. Add php extension to php.ini

[rdkafka]
extension=rdkafka.so

3. Install package for symfony:

composer req symfony/messenger enqueue/rdkafka enqueue/enqueue-bundle sroze/messenger-enqueue-transport enqueue/async-event-dispatcher

4. Register bundles - add to config/bundles.php

Enqueue\Bundle\EnqueueBundle::class => ['all' => true],
Enqueue\MessengerAdapter\Bundle\EnqueueAdapterBundle::class => ['all' => true],

5. add file config/packages/enqueue.yaml:

enqueue:
  default:
    transport:
      dsn: "rdkafka://"
      global:
        group.id: 'myapp'
        metadata.broker.list: "%env(KAFKA_BROKER_LIST)%"
      topic:
        auto.offset.reset: beginning
      commit_async: true
    client: ~

6. add file config/packages/messenger.yaml:

framework:
  messenger:
    failure_transport: failed

    transports:
      async:
        dsn:  "%env(MESSENGER_TRANSPORT_DSN)%"
      failed:
        dsn: "doctrine://default?queue_name=failed"

    routing:
      'App\Message\EmailNotification': async

7. add to .env:

###> messenger ###
MESSENGER_TRANSPORT_DSN=enqueue://default?topic[name]=messages&queue[name]=messages
KAFKA_BROKER_LIST=node-1.kafka.host:9092,node-2.kafka.host:9092,node-3.kafka.host:9092
###< messenger ###

8. Message and MessageHandler from documentation: https://symfony.com/doc/current/messenger.html

9. Run consumer:

php bin/console messenger:consume async

Good luck!

Gayan
  • 3,614
  • 1
  • 27
  • 34
Arkadiusz G.
  • 1,024
  • 10
  • 24
  • It is so hard setting up kafka with symfony messenger. I have been following your guidelines and through different articles. I would say your guideline is the most closes I got making things work with kafka + symfony. I have done everything, including step 8. but I still can't manage to produce and consume successfully. There is no errors following your steps and no error consuming or producing, but I don't see any messages on confluent.io kafka server. Can you share your setup with us, I feel like there must be something we are missing. – Basit Mar 05 '20 at 11:43
  • after creating a topic in the kafka interface from confluent. I have manage to produce messages, but I still cant consume messages with `php bin/console messenger:consume async` - any help will be appreciated – Basit Mar 05 '20 at 12:14
  • 2
    `MESSENGER_TRANSPORT_DSN=enqueue://default?topic[name]=messages&queue[name]=messages` and make sure you create topic messages - everything should work fine after that, it took me lots of debugging to find out that :) – Basit Mar 07 '20 at 21:53