13

Synchronous logging incurs a large performance penalty as it may block. Is there a standalone Ruby library that does asynchronous logging (log4r doesn't seem to)? Can I modify the standard library logger to log asynchronously? I'm looking for something like log4j's AsyncAppender - but preferably an implementation that makes use of Ruby's code blocks to shift as much work to the background thread as possible.

Nakilon
  • 34,866
  • 14
  • 107
  • 142
Nicholas White
  • 2,702
  • 3
  • 24
  • 28
  • Are `ruby` interpreters these days still fairly ham-strung when it comes to multi-threaded code? Is there still a giant interpreter lock that essential single-threads native Ruby code? I'm curious because, unless you're already using a framework like [eventmachine](http://rubyeventmachine.com/) to force _all_ your IO to be async, you might not actually see any benefit from making just the logging async. – sarnold Jun 27 '11 at 22:33
  • I usually use JRuby, so have real Java threads (I think)... – Nicholas White Jun 27 '11 at 22:39
  • ooh ;) that's a compelling reason to consider using JRuby, then. Thanks. – sarnold Jun 27 '11 at 22:40
  • 2
    The GIL is often misunderstood— if one thread is waiting on IO, Ruby does switch to another thread. So there is a concurrency benefit to using a threaded logger. – John Bachir Nov 04 '11 at 01:18

4 Answers4

17

I know you shouldn't really answer your own question, but it seems everything is easy in ruby:

require 'thread'
require 'singleton'
require 'delegate'
require 'monitor'

class Async
  include Singleton

  def initialize
    @queue = Queue.new
    Thread.new { loop { @queue.pop.call } }
  end

  def run(&blk)
    @queue.push blk
  end
end

class Work < Delegator
  include MonitorMixin

  def initialize(&work)
    super work; @work, @done, @lock = work, false, new_cond
  end

  def calc
    synchronize {
      @result, @done = @work.call, true; 
      @lock.signal
    }
  end

  def __getobj__
    synchronize { @lock.wait_while { !@done } }
    @result
  end
end

Module.class.class_exec {
  def async(*method_names) 
    method_names.each do |method_name|
      original_method = instance_method(method_name)
      define_method(method_name) do |*args,&blk|
        work = Work.new { original_method.bind(self).call(*args,&blk) }
        Async.instance.run { work.calc }
        return work
      end
    end
  end
}

And for my logging example:

require 'Logger'
class Logger
  async :debug
end
log = Logger.new STDOUT
log.debug "heloo"

As return values work, you can use this for just about anything:

require "test/unit"
class ReturnValues < Test::Unit::TestCase
  def do_it
    5 + 7
  end
  async :do_it
  def test_simple
    assert_equal 10, do_it - 2
  end
end
Nicholas White
  • 2,702
  • 3
  • 24
  • 28
  • Excellent. Ruby's expressive nature constantly amazes me. You mentioned using JRuby so you potentially have real threads but you're also using a lot of dynamic ruby magic so how well does JRuby cope with all the dynamic tricks? It'd be cool to see some benchmarks with JRuby and MRI side by side. – David K. Jul 02 '11 at 03:55
  • Nice code! You might be better off using fibers/eventmachine. Threads will do the job, but they are much more heavyweight in MRI (and in the future in JRuby) and require management and synchronization. I think there is already an event machine file streamer which might be close to what you want: http://stackoverflow.com/questions/2749503/what-is-the-best-way-to-read-files-in-an-eventmachine-based-app – Greg Weber Jul 02 '11 at 18:53
2

No personal experience with that:

The Swiftcore Analogger implements a fast asynchronous logging system for Ruby programs as well as client library for sending logging messages to the Analogger process.

Analogger will accept logs from multiple sources and can have multiple logging destinations. Currently, logging to a file, to STDOUT, or to STDERR is supported. A future revision may support logging to a database destination, as well.

Analogger depends on EventMachine (http://rubyforge.org/projects/eventmachine) to provide the framework for the network communications, though EM is not used for the client library.

miku
  • 181,842
  • 47
  • 306
  • 310
  • This seems to do the asynchronous logging in a separate process entirely - I'd prefer in-process logging (just on a different thread) – Nicholas White Jun 27 '11 at 22:42
1

The built in Logger class is already thread safe

Eric Hodel
  • 19
  • 1
  • I want the log messages to be written to disk on a separate thread, so if the IO blocks it doesn't slow down my main thread. – Nicholas White Aug 05 '14 at 09:16
0

Checkout Dunder https://github.com/fonsan/dunder

I created this gem 6 months ago that does just this and more

Fonsan
  • 121
  • 2