Which is the most performance based method of writing logs for Application

Question

I am working in a client server application in which multiple clients and server are working on socket based communication for financial transactions where performance is very critical. Currently I am using streamwriter of system.IO namespace to write logs in file. For a single transaction I need to call streamwriter method 50 times to log different value and for more than 50,000 transactions, time taken for this logging become very important.

How can I reduce time taken by application to do logging? Whether I need to choose some other approach or any other class instead of streamwriter? What will be the best way to do logging with lesser time.

Answer 1

If performance is key, then I would consider looking at Event Tracing for Windows (AKA ETW).

With .NET 4.5 and the introduction of the EventSource class this has made ETW much easier to implement than in the past.

Vance Morrison's Weblog has some good articles on this topic.

For an overview of the architecture see Improve Debugging And Performance Tuning With ETW.

There is also the Semantic Application Block from Microsoft's patterns & practices team that makes it easier to incorporate the EventSource functionality and to manage logging behavior.

Answer 2

I suggest you to try Log4Net, you can configure where (file, database, xml) and when (bath, transaction, ...) and easily switch tracing level (debug, info, warning, ...)

Writing a log system from scratch is not worth it.

Answer 3

Cache the values before writing them to disk.

Only commit the log when you have finished your transaction.

Something like this:

StringBuilder builder = new StringBuilder();
// do transaction step 1
builder.Append("Transaction step 1" + environment.NewLine); // or however you add a line to you log

// step 2
builder.Append("Transaction step 2" + environment.NewLine);

//...

// step 50
builder.Append("Transaction step 50" + environment.NewLine);

// now write to the file
File.WriteAllText(@"C:\log.txt", builder.ToString());

You can add in some handling if there is an error in any step to still write the log.

You could also use some opensource tool like log4net: http://logging.apache.org/log4net/.

Answer 4

I suggest log into database (high perfomance, maybe embedded sqlite/sqlce). Bonus - you can structure and query your log entries.

Answer 5

To reduce the time taken by logging I suggest:

• Ensure that the data being logged requires minimal conversion/formatting
• Create or use a logging library that:
  • When called puts the logging data (along with time, thread id, and other tags to be logged) in a buffer.
  • Periodically flushes the buffered data to disk (i.e. immediately when the buffered data is big enough to fill at least one physical block in the log file, or immediately when the system becomes idle, or periodically every x seconds).
  • Locks the log file for exclusive write access, so you can view it while the software is running but other processes can't lock it under your feet.
  • Uses a separate thread to handle the flushing i.e. don't slow down your worker threads.
• If you have many server processes, consider using IPC to send the log data to a single point, to minimise the number of active files being written and number of buffers in use (you may have to test so see if this is worth it, and you may have to add tags to show the source of each entry).
• Do scheduled/idle time backups of the logs to prevent them getting too big.

Answer 6

One way to improve performance of you application and be able to write out all those log messages is to queue them up in an MSMQ queue, and have a windows service application process the queued log messages when the server is not under load. You could have the queue on a completely separate server too.

Implementation-wise you can setup a WCF web service that uses MSMQ for processing your log messages. That makes it easier than having to setup a windows service.