I am reading a CSV file line by line. I will process the line and then dump it into another CSV file (simplified version of the problem). I am using MSMQ as a "fire and forget" mechanism. However, it's significantly slower than expected (more than 100%). Is there something I am missing here, like a setting?
I tried to run the Send call in a seperate thread. It didn't help.
Sender code:
class Program
{
static void Main(string[] args)
{
string queueName = ".\\Private$\\TestQueue";
if (!MessageQueue.Exists(queueName))
{
MessageQueue.Create(queueName, false);
}
MessageQueue messageQueue = new MessageQueue(queueName);
string path = @"file.csv";
int lines = 0;
Stopwatch stopwatch = new Stopwatch();
stopwatch.Start();
using (StreamReader reader = new StreamReader(path))
{
string readLine;
while ((readLine = reader.ReadLine()) != null)
{
lines++;
messageQueue.Send(new Message(readLine));
}
}
stopwatch.Stop();
Console.WriteLine("Lines: " + lines);
Console.WriteLine($"Time: {stopwatch.Elapsed.ToString("g")}");
}
}
Receiver code:
class Program
{
static void Main(string[] args)
{
MessageQueue queue = new MessageQueue(".\\Private$\\TestQueue");
queue.Formatter = new XmlMessageFormatter(new[] { typeof(string) });
WriteToReport(queue);
}
private static void WriteToReport(MessageQueue queue)
{
StreamWriter writer = new StreamWriter($@"Log_{DateTime.Now:yyyyMMddhhmmssms}.csv");
while (true)
{
Message message = queue.Receive();
string row = message?.Body as string;
if(message == null || string.IsNullOrEmpty(row)) break;
writer.Write(row);
}
writer.Close();
}
}
Note:
I tried to just read the messages in the receiver program without writing them to anywhere. It's still slow.
Sending the messages in a seperate thread did not help.
Increasing the queue size didn't help.
Shouldn't the Send method be asynchronous? I was expecting it to just Send the message and continue processing.