3

I'm taking over a Ruby on Rails site and I'm discovering that the site has huge performance issues. Sometimes the site doesn't even load. And this is not new for the site.

It's on a Rackspace Server (First Generation) with 2gb Ram.

It's running on Ubuntu Hardy using Apache2 and MySQL. The RoR site is running an older version of Ruby (1.8.7) and Rails (3.2.1).

According to top (shift m), Apache (res column) uses about 6mb to 10mb per process.

It's using prefork mpm with the below specs: StartServers 2 MinSpareServers 2 MaxSpareServers 2 MaxClients 50 MaxRequestsPerChild 100

And Passenger is set to have: PassengerMaxPoolSize 10

Passenger-memory-stats show that Rails uses, on average, about 40mb.

free -ml consistently shows 100mb to 1500mb of free memory.

Passenger-status sometimes shows as high as 250 on the waiting on global queue but generally varies from 0 to 100.

I have tried playing around with MaxClients and the PoolSize but eventually the site succumbs to slowness or simply being unaccessible at some point but may, I'm assuming when traffic eases up, load fine again at a later point.

Loading the actual Rails sites can sometimes takes forever but loading static files (images, txt files) works fine. Although sometimes it gets to the point where you can't even load static files.

Any pointers on trying to get this working? For the amount of traffic it gets (about 250k impressions per month) it seems like the server should be fine for this site.

Edit: I responded to comments about though I put it in here anyway.

Database is about 1gb in size. There are quite a bit of spam issues (new accounts that are obvious spam which average about 1k per day, spam posts/comments, etc). Mysql-slow.log shows nothing so far.

Thanks for all the comments. I had hoped that it was simply me being an idiot on the Apache or Passenger server settings. My next steps is to start investigating code and queries.

Singraye
  • 31
  • 1
  • 3
  • Are you working with lots of data? Or intensive queries? Is it just as slow with an empty database? – Sam Starling Aug 04 '14 at 15:47
  • What pages seem to cause the biggest issue? Have to analyzed the queries that are running? I know I have experienced huge performance impacts with `n+1` queries for nested resources. I just had to completely refactor multiple controllers and models to alleviate these issues which do not occur in a small development environment because the resources are much smaller – engineersmnky Aug 04 '14 at 16:11
  • 2
    Start investigating in the app itself to find out what it's doing. I guess the response time is low. This is why the passenger queue is stacking up. – iltempo Aug 04 '14 at 16:14
  • Investigating the app and queries is my next step but I hoped that it might have just been some apache or passenger setting that I messed up on. I have enabled the slow mysql log but it's empty. I'm not sure if it's a single specific page causing issues or not. When it slows down, the entire site slows down. Sometimes it slows down to the point that loading static files doesn't work anymore (apache crash?). But otherwise, the site is pretty snappy and loads times are fine. Any pointers on trying to figure out what page is causing this? – Singraye Aug 04 '14 at 16:50

2 Answers2

7

Without knowing much I can only give more general advice. For improving site performance remember the Perfomance Golden Rule:

80-90% of the end-user response time is spent on the front-end. Start there.

Below is a non-exhaustive list areas of improvement for increasing performance in a Rails app:

Diagnosing a Slow Rails App:

YSlow

A useful diagonsis tool for identifying perfomance issues is Yslow. It's a browser extension that diagnoses and identifies common issues slowing down your app (particularly on the front end).

Back-end Tools

For your Rails back-end I recommend incorporating tools such as Bullet & NewRelic directly into your development processes, so that while you're developing you can spot bad queries immediately while they are still easy to fix.

Check Server Console Logs

Checking your server logs is an effective method for diagnosing what components of your Rails app is taking the longest. E.g. below are sample logs from two unrelated production Rails apps running in my local development environment (with the huge database i/o portion excluded) :

# app1: slowish
  Rendered shared/_header.html.erb (125.9ms)
  Rendered clients/details.html.erb within layouts/application (804.6ms)
  Completed 200 OK in 3655ms (Views: 566.9ms | ActiveRecord: 1236.9ms)

# app2: debilitatingly slow
  Rendered search/_livesearch_division_content.js.erb (5390.0ms)
  Rendered search/livesearch.js.haml (34156.6ms)
  Completed 200 OK in 34173ms (Views: 31229.5ms | ActiveRecord: 2933.4ms)

App1 & App2 both suffer from performance issues, but App2's performance issues are clearly debilitatingly slow. But with these server logs, I know for App1 that I should look into clients/details.html.erb, and that for App2 I absolutely need to investigate search/livesearch.js.haml. (I discovered from looking that App2 had uncached, large N+1 queries over lots of data -- I'll touch on this later)

Improving Front-end Performance

Budget your page size strictly

To maintain fast load times you need reduce the amount/size of your page assets (JS/CSS/Images). So think about your page size like a budget. For example, Hootesuite recently declared that their home page now has a strict page-size budget of 1 mb. No exceptions. Now check out their page. Pretty fast isn't it?

Easy wins for reducing your page size include stripping out unused JS or CSS files, including them only where needed, and changing static images into much smaller vectors.

Serve smaller image resolutions based on screen width

Image loading is a large cause of slow page loading times. A large 5mb image used in the background of your splash page can easily be brought down to 200kb-400kb in size, and still be high quality enough to be hardly indistinguishable from the higher resolution original. The difference in page load times will be dramatic.

You should do the same with uploaded images as well. E.g. if a user uploads a 5mb image for his banner or avatar, then it's essential that you serve this uploaded image at lower file sizes/resolutions depending on the size that it will be displayed. Carrierwave, Fog, combined with rmagick OR minimagic is a popular technique used with Amazon S3 to achieve better image uploading/resizing. With them, you can dynamically serve smaller image resolutions to fit the screen size of your user. You can then use media queries to ensure that mobile devices get served smaller image resolutions & file sizes than desktops with Retina screens.

Use a Content Delivery Network to speed up asset loading

If your site deals with lots of images or large files, then you should look into using a Content Delivery Network (CDN's) such as Cloudfront to speed up asset/image loading times. CDN's distribute your assets files across many servers located around the world and then use servers closest to the geographical region of the user to serve the asset. In addition to faster speeds, another benefit of a CDN is that it reduces the load on your own server.

Fingerprint Static Assets

When static assets are fingerprinted, when a user visits your page their browser will cache a copy of these assets, meaning that they no longer need to be reloaded again for the next request.

Move Javascript files to the bottom of the page

If there are other javascript files included without using the Rails way, then be aware that if javascript assets are placed on the top of the page, then a page will remain blank as its loading while a user's browser attempts to load them. Rails will automatically place javascript files to the bottom of your page if you use the asset pipeline or specify javascript files using the javascript_include_tag to prevent this problem, but if that's not the case make sure that external Javascript files are included at the bottom of the page.

Improving Back-end Performance

Cache, Cache, Cache (with Memcache/Dalli)!

Among backend performance optimizations, there is no single performance enhancement that can come even close to matching the benefits provided with caching. Caching is an essential component of pushing any Rails app to high scalability. A well implemented caching regime greatly minimizes your server's exposure to inefficient queries, and can reduce the need to undergo painful refactorings of existing non-performant code.

For example, with my App2 example mentioned above, after we implemented a simple page cache, our 34 second page load time dropped down to less than a second in production. We did not refactor a single line of this code.

Page content that is accessed frequently, yet change relatively infrequently are easiest to cache and benefit most from caching. There multiple ways to cache on the serverside, including page caching and fragment caching. Russian doll caching is now the favoured fragment caching technique in Rails. Here's a good place to start.

Index (Nearly) Everything

If you are using SQL for your database layer, make sure that you specify indexes on join tables for faster lookups on large associations used frequently. You must add them during migrations explicitly since indexing is not included by default in Rails.

Eliminate N+1 queries with Eager Loading

A major performance killer for Rails apps using relational (SQL) databases are N+1 queries. If you see in your logs hundreds of database read/writes for a single request in a way that resembles a pattern, then it's likely you already have a serious N+1 query issue. N+1 queries are easy to miss during development but can rapidly cripple your app as your database grows (I once dealt with an that had twelve N+1 queries. After accumulating only ~1000 rows of production data, some pages began taking over a minute to load).

Bullet is a great gem for catching N+1 queries early as you develop your app. A simple method for resolving N+1 queries in your Rails app is to eager load the associated Model where necessary. E.g. Post.all changes to Post.includes(:comments).all if you are loading all the comments of each post on the page.

Upgrade to Rails 4 and/or Ruby 2.1.x or higher

The newer version of Rails contains numerous performance improvements that can speed up your app (such as Turbolinks.) If you are still running Rails 3.2.1 then you need to upgrade to at least Rails 3.2.18 for security reasons alone.

Ruby 2.1.x+ contain much better garbage collection over older versions of Ruby. So far reports of people upgrading have found notable performance increases from upgrading.


These are a few performance improvements that I can recommend. Without more information I can't be of much help.

Community
  • 1
  • 1
Kelsey Hannan
  • 2,857
  • 2
  • 30
  • 46
0

Index only the models that are accessed normally. That should give you a significant performance boost as well.

user3646743
  • 159
  • 2
  • 12