1

I've always thought that 1.9.x was supposed to be faster than REE, but consuming more memory. But as it was recently pointed out to me, REE is actually faster than executing RSpec specs, and by a huge margin. 

$ rvm use ree
$ rvm gemset create sandbox
$ rvm gemset use sandbox
$ gem install rspec

$ time rspec foo_spec.rb 
No examples found.

Finished in 0.07346 seconds
0 examples, 0 failures

real    0m0.104s
user    0m0.059s
sys 0m0.015s

and after doing the same thing with 1.9.3, I'm getting

$ time rspec foo_spec.rb 
No examples found.

Finished in 0.13922 seconds
0 examples, 0 failures

real    0m0.208s
user    0m0.122s
sys 0m0.022s

That's twice as much with an empty gemset, containing only rspec and executed on a empty spec file. I'm seeing even larger differences on gemsets containing multiple gems.

Why is this happening, isn't 1.9.3 supposed to be the fastest version currently available?

I'm running the latest versions installed via RVM on OS X Lion.

Holger Just
  • 52,918
  • 14
  • 115
  • 123
Jakub Arnold
  • 85,596
  • 89
  • 230
  • 327
  • 2
    You are only measuring startup time, not actual speed at useful tasks. Chose something more complex which matches your code instead to get useful benchmark results. – Holger Just Dec 18 '11 at 00:37
  • Also, beware of file-system caching. To get more accurate results, run each test, say, 20 times consecutively and take the median of the last 15 runs. – phs Dec 18 '11 at 00:42
  • @phs yes I'm aware of the file-system caching, but the problem is when the tests get run for the first time, since I'm only going to run them again when I change the files – Jakub Arnold Dec 18 '11 at 02:11
  • @HolgerJust I tried the same thing even on more complex scenarios and it always turns out that REE runs the specs faster. The example I posted is just illustrative. I even tried completely reinstalling 1.9.3 to have a clean start. – Jakub Arnold Dec 18 '11 at 02:13

2 Answers2

5

It depends on what you bench against really. If you do something as simple as that then it doesn't even count in my opinion. So, here's an example of my Backup gem which does a lot of file "requires" and runs a bunch of real world specs.

Ruby Enterprise Edition

time bundle exec rspec spec

real    0m5.579s
user    0m3.427s
sys 0m0.465s

502 examples, 0 failures

Ruby 1.9.3p0

bundle exec rspec spec

real    0m3.863s
user    0m3.552s
sys 0m0.299s

502 examples, 0 failures

The start-up time is generally faster with 1.9.3p0 since a certain patch made it in that changes the algorithm of the file require function, which on it's own I believe reduced the average load time of apps similar in size of average Ruby on Rails applications by about 30%. But, if there are barely any files initially loaded, and the process doesn't run for long, then it might be slower.

Michael van Rooijen
  • 6,683
  • 5
  • 37
  • 33
1

I would guess that you're seeing the results of GC performance tuning which are really still in REE's favour (based on real-world benchmarking we've done with a large Rails app under both REE and 1.9.3).

To eliminate GC overhead from your benchmarking, so you're really comparing apples to apples, disable it as early as possible in the process. You can do that as follows:

GC.disable

Of course, you'll only get useful results if you have sufficient memory to run all the tests without invoking GC; if your machine starts swapping then the results won't be useful.

kranzky
  • 1,328
  • 11
  • 16
  • You definitely need to turn your GC - try running this before you run your specs: export RUBY_HEAP_MIN_SLOTS=600000 && export RUBY_GC_MALLOC_LIMIT=59000000 && export RUBY_FREE_MIN=200000 – Michael Oct 31 '12 at 04:11