Publish multiple messages to RabbitMQ from a file

Question

Publishing single messages to a RabbitMQ queue can be easily done with the UI, by simply putting the message in the UI and clicking the "Publish Message" button.

How do you publish a batch of messages?

I have a file with messages to be sent to RabbitMQ. Each line has one message.

How can I publish all the messages from the file to my RabbitMQ server?

Is there a way to do it from command line?

score 21 · Answer 1 · answered Sep 15 '16 at 12:29

21

Using rabbitmqadmin

while read -r line; do 
  echo $line | rabbitmqadmin publish exchange=amq.default routing_key=my_queue ; 
done < messages

Not specifying the payload parameter to rabbitmqadmin publish means it reads the payload from stdin.

answered Sep 15 '16 at 12:29

looseend

261
2
6

Oleg SH · Answer 2 · 2018-05-17T09:23:51.693

15

you can use curl and rabbitmq api:

curl -u login:pass -i -H "content-type:application/json" -X POST http://localhost:15672/api/exchanges/%2Fvhost/exchange/publish -d'{"properties":{},"routing_key":"","payload":"you message","payload_encoding":"string"}'

this is an example of one message, by analogy, you can write a script

edited May 17 '18 at 09:23

answered May 17 '18 at 08:45

Oleg SH

443
5
11

score 7 · Answer 3 · answered Nov 02 '15 at 10:27

7

You need to use rabbitmqadmin cli tool:

https://www.rabbitmq.com/management-cli.html

rabbitmqadmin publish exchange=amq.default routing_key=test payload="hello, world"

answered Nov 02 '15 at 10:27

aeryaguzov

1,143
1
10
21

1

How does this snippet publish the messages/payloads from the file? – summerbulb Nov 02 '15 at 12:11
try smth like this: `while read line; do rabbitmqadmin publish exchange=amq.default routing_key=test payload="$line"; done < file` – aeryaguzov Nov 02 '15 at 13:48
1

`payload="$line"` would break JSON \-escapes like `{"foo": "\"nested quotes\""}` (because you shell also interprets \-escapes). But you can pass payload to rabbitmqadmin's stdin. – Boris Berkgaut Apr 06 '16 at 14:37
1

Yeah, as noted it's unfortunate that `rabbitmqadmin` requires input to be text (due to input from the shell), not to mention that it uses the http api for publishing/consuming so it tends to be pretty slow for larger usage. – joekarl Oct 17 '16 at 14:29

score 5 · Answer 4 · answered Mar 28 '19 at 18:40

As a variation to the answer from looseend you could also use GNU Parallel

This will yield far better performance if you have a large file.

  cat messages | parallel -j 100 \
  ./rabbitmqadmin -H $RABBITMQ_HOST \
                  -u $RABBITMQ_USERNAME \
                  -p $RABBITMQ_PASSWORD  \
                  publish exchange=amq.default \
                  routing_key=myqueue \
                  payload="{}"

This will run with 100 jobs. Omit host and credentials if not needed.

With this it went 5 time quicker :) – glmrenard Jul 18 '19 at 13:48 — glmrenard, Jul 18 '19 at 13:48

score 1 · Answer 5 · answered Jun 09 '16 at 08:15

I've updated rabbitmqadmin file to support file content publishing. Try to find line containing EXTRA_VERBS = { as well as def invoke_publish(self): and update their related code as follows

EXTRA_VERBS = {
    'publish': {'mandatory': ['routing_key'],
                'optional':  {'payload': None,
                              'pfile': None,
                              'properties': {},
                              'exchange': 'amq.default',
                              'payload_encoding': 'string'},
                'json':      ['properties'],
                'uri':       '/exchanges/{vhost}/{exchange}/publish'},
    'get':     {'mandatory': ['queue'],
                'optional':  {'count': '1', 'requeue': 'true',
                              'payload_file': None, 'encoding': 'auto'},
                'uri':       '/queues/{vhost}/{queue}/get'}
}

and

def invoke_publish(self):
    (uri, upload) = self.parse_args(self.args, EXTRA_VERBS['publish'])
    if not 'payload' and 'pfile' in upload:
        data = sys.stdin.read()
        upload['payload'] = b64(data)
        upload['payload_encoding'] = 'base64'
    elif not 'payload' in upload:
        with open('populate/' + upload['pfile']) as f: data = f.read()
        upload['payload'] = b64(data)
        upload['payload_encoding'] = 'base64'
    resp = json.loads(self.post(uri, json.dumps(upload)))
    if resp['routed']:
        self.verbose("Message published")
    else:
        self.verbose("Message published but NOT routed")

Remove 'populate/' + from the following line if you want to provide file using absolute path.

with open('populate/' + upload['pfile']) as f: data = f.read()

Without updates to open(...) following command worked for me fine, assuming that file rules.json was placed in relative directory "populate"

python rabbitmqadmin.py publish exchange=feed-mgmt-in routing_key='#' properties='{"type":"domain-collections/rules"}' pfile="rules.json"

score 0 · Answer 6 · answered Apr 26 '23 at 23:11

https://github.com/selency/amqp-publish can be used for publishing RabbitMQ messages to the standard 5672 port even in case admin port 15672 is not enabled.

# cd /tmp
# curl -vL https://github.com/selency/amqp-publish/releases/download/v1.0.0/amqp-publish.linux-amd64 -o amqp-publish
# chmod +x ./amqp-publish

# ./amqp-publish --uri="amqp://admin:password@localhost:5672/" --exchange="foo" --routing-key="awesome-routing-key" --body="hello, world!"

Though it didn't work for me silently from within k8s pod's shell... — Vadzim, Apr 26 '23 at 23:47

Vadzim · Answer 7 · 2023-04-27T00:00:06.253

amqp-publish from https://github.com/alanxz/rabbitmq-c can be used for publishing RabbitMQ messages to the standard 5672 port even in case admin port 15672 is not enabled.

Manual install inside k8s pod:

# cd /tmp
# curl -LO http://mirror.centos.org/centos/7/os/x86_64/Packages/librabbitmq-0.8.0-3.el7.x86_64.rpm
# curl -LO http://mirror.centos.org/centos/7/os/x86_64/Packages/librabbitmq-examples-0.8.0-3.el7.x86_64.rpm
# yum install librabbitmq-tools-0.11.0-5.el9.x86_64.rpm
# yum install librabbitmq-examples-0.8.0-3.el7.x86_64.rpm

Example message publishing command that worked for me:

# amqp-publish --url="amqp://usr:pwd@host:5672" -e exchange-name -r routing-key <message-body.json

Example command to consume the response queue:

# amqp-consume --url="amqp://usr:pwd@host:5672" -q queue-name cat

score -2 · Answer 8 · answered Nov 04 '15 at 11:57

-2

Just use amqp-tool:

npm install amqp-tool -g

amqp-tool --host rabbitmq.local -u <user> -p <password> -q <queue name> --import file.json

answered Nov 04 '15 at 11:57

matagus

6,136
2
26
39

I tried but get (node:24786) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added. Use emitter.setMaxListeners() to increase limit – glmrenard Jul 18 '19 at 13:47
I also tried but its giving error for routingKey even routing key is provided. – Hus Mukh Nov 29 '20 at 02:51

Publish multiple messages to RabbitMQ from a file

8 Answers8