2

I've created a client of Apache Kafka using Kafkajs, and I'm trying to read messages from a topic in Kafka. It's working fine if I console.log(message). But I want to send message to client everytime whenever a new message is produced/written in the topic, the consumer is listening to, from producers, while keeping the connection alive.

 // function, which is being called whenever it's specified route is being requested
 async readMessage(req, res, next, consumer) {
    const resMessage = {};

    res.writeHead(200, {'Content-Type': 'text/plain'});

    await consumer.run({
        eachMessage: async ({ topic, partition, message }) => {  
            res.write(message.value.toString());
        },
    });

    // res.send(resMessage);
}

But after I send data to the express.js server, res.write() doesn't send the data to the client (I'm using Postman as my Node.js client). How do I flush the data written in res.write(), before res.end() is invoked?

S. Joshi
  • 153
  • 1
  • 3
  • 14
  • Last time I traced through this in the debugger, it appeared to me that `res.write()` does send the data (no buffering by Express or node.js). It may be buffered briefly by the OS in service of Nagle's algorithm. The last time I looked into this, the issue was that the client was not making the data available to the recipient code until the end of the request was received. You don't say anything about what you're trying to do on the client end or how that code works, but that seems like a possibility in your case. You could know for sure if you installed some sort of network spy. – jfriend00 Feb 24 '20 at 09:09
  • @jfriend00 I'm sending request to the route I created in my nodejs server through Postman, and It did received data, but didn't show it. How can I resolve that? I want client to show every new message Producer API send to the kafka topic. I've not coded any client, it's just a url being requested by the postman. – S. Joshi Feb 24 '20 at 10:43

1 Answers1

3

res.write() does send the data (no buffering by Express or by nodejs) as soon as you call it. I've traced through it in the debugger and seen it send the data without delay. It may be buffered briefly by the OS in service of Nagle's algorithm, but that would only be a very short delay (milliseconds). So, the data is getting sent to the client. It is much more likely that your issue is in the http client that is receiving the data.

Most http clients are built to expect request/response. Send a request, wait for the entire response, then notify the caller. So, if you want to get regular data from the http stream as it arrives, you will need a non-traditional http client that will notify you when data arrives like that. You will probably also need to invent some sort of protocol that allows the client to know when a full piece has arrived since packets can be chunked or grouped in random ways.

What might be easier than using http for this is to use an actual message-based protocol such as webSockets or socket.io that is explicitly designed for the exact kind of thing you're trying to do. The client would establish a connection to the server and then the server is free to send messages to the client at any time. webSocket or socket.io are message-based so they already do all the work for you of delineating a message, packing it together, delivering it, unpacking it and notifying the recipient that a message has arrived. This is exactly what webSocket/socket.io were designed for.

For one-way message sending (from server to client), you could also use Server Sent Events which is an extension of http.

jfriend00
  • 683,504
  • 96
  • 985
  • 979
  • Makes sense. Thanks! – S. Joshi Feb 25 '20 at 02:30
  • 2
    I'm making raw HTTP requests by hand using `nc -v localhost 3000`, and when I call `res.flushHeaders()` the headers are flushed, but `res.write()` never writes chunks at all until I call `res.end()`. If I write `10_000_000` lines of output as a response, it never flushes any data and the server crashes OOM. – aradil Jul 24 '20 at 14:47
  • @aradil - Well, `res.write()` returns a boolean related to flow control. You have to pay attention to that if you're writing lots of data. For any further help, we'd have to see your actual code (in your own question). – jfriend00 Jul 24 '20 at 14:50
  • 1
    This was my mistake -- I suspect that others might be making the same mistake as well. It was not flushing the buffer because the event loop was pegged in my response until OOM; when I changed my code to write to the response on an interval, the buffer was flushed. – aradil Jul 24 '20 at 15:09
  • 1
    @aradil - Well, really you just need to hook in logic for the `drain` event. When `res.write()` returns `false` (buffer is full), you have to suspend writing until you get a `drain` event. That is the designed way to send a lot of data. You don't have to use an interval. – jfriend00 Jul 24 '20 at 15:54