Problem: Can't figure out how to make an output csv in the below mentioned architecture.
Approach: I have a Producer which actually validate the CSV's data, prepare the data for consumer and push the data into the message queue. Here I am using RabbitMQ as message queue and it is to be implemented in Node.js. Now, when the producer sends the data in the message queue after the producer is not getting any of response from the consumer if the data is being successfully processed by the consumer or not. Here Processing Huge CSV File using Producer - Consumer Pattern, I can see, Yogendra is facing the same issue (not exactly) but in Java.
CSV data have OrderID , OrderAddress ,OrderDate
Output CSV expectation : OrderID , OrderAddress ,OrderDate , Status, Remarks
where Status have true , false value and Remarks will have failure reason if any.
Here is the code of Producer and Consumer :
Producer.js
try {
const connection = await amqp.connect('amqp://localhost');
const channel = await connection.createChannel();
const exchangeName = 'csv_exchange';
const routingKey = 'csv_data';
const csvFilePath = __dirname + "/test.csv";
console.log("csvFilePath",csvFilePath)
await channel.assertExchange(exchangeName, 'direct', { durable: true });
let csvData = await readCSVFile(csvFilePath);
console.log("csvData", csvData, typeof csvData)
csvData = JSON.parse(csvData);
for (let i =0 ; i< csvData.length ; i++) {
const message = JSON.stringify(csvData[i]);
let publishedData = channel.publish(exchangeName, routingKey, Buffer.from(message));
console.log(`Message published: ${message}`);
}
await channel.close();
await connection.close();
} catch (error) {
console.error('Error in publishMessages:', error);
}
}
Consumer.js
try {
const connection = await amqp.connect('amqp://localhost');
const channel = await connection.createChannel();
const exchangeName = 'csv_exchange';
const queueName = 'csv_queue';
const routingKey = 'csv_data';
await channel.assertExchange(exchangeName, 'direct', { durable: true });
await channel.assertQueue(queueName, { durable: true });
await channel.bindQueue(queueName, exchangeName, routingKey);
channel.consume(queueName, async (message) => {
// console.log(Buffer.to(message))
const row = JSON.parse(message.content.toString("utf8"));
console.log('Received message:', row);
// Process the row and save it to MongoDB
// await saveToMongoDB(row);
channel.ack(message);
});
} catch (error) {
console.error('Error in consumeMessages:', error);
}
}