NodeJS API with Kafka back-end

Question

I want to build a NodeJS based API that is backed with a pub-sub paradigm with e.g. Kafka. Here is a skeleton of what I want to do.

const express = require('express')
const serverApp = express()
serverApp.get('/book/:bookId', (req, res) => {
    producer.send(JSON.stringify({
        action: 'get',
        message: req.params.bookId
    }))
    consumer.on('message', (data) => {
        res.status(200).send(JSON.parse(data))
    })
})

With the option above, the first invocation works but, the subsequent ones keep failing with ERR_HTTP_HEADERS_SENT.

Keeping the consumer.on outside of serverApp.get will need to have the req and res co-ordinated.

How do I implement such an API?

You'll need to unsubscribe every time... – Sandesh K Feb 21 '20 at 12:21 — Sandesh K, Feb 21 '20 at 12:21

Yaroslav Gaponov · Accepted Answer · 2020-02-21T13:12:48.667

4

For example as skeleton

const express = require('express')
const serverApp = express()

const responses = Object.create(null);

consumer.on('message', (data) => {
    const result = JSON.parse(data)
    const res = responses[result.replyId]
    res.status(200).send(result)
});

serverApp.get('/book/:bookId', (req, res) => {

    const replyId = Math.random().toString(36).substr(2);    
    responses[replyId] = res;

    producer.send(JSON.stringify({
        action: 'get',
        replyId,
        message: req.params.bookId
    }))
})

edited Feb 21 '20 at 13:12

answered Feb 21 '20 at 12:38

Yaroslav Gaponov

1,997
13
12

1

While I try this, `sunstr` should be `substr` and `replyId` should have a key? – cogitoergosum Feb 21 '20 at 13:04
`replyId` is just some unique id. Remove handler must return it back. – Yaroslav Gaponov Feb 21 '20 at 13:16
Yes, but, in your `JSON.stringify`, the `replyId` should be key-value pair, right? – cogitoergosum Feb 21 '20 at 13:23

NodeJS API with Kafka back-end

1 Answers1