-1

Until now, our site has had a modest amount of traffic. None of our developers are big ops guys, but we've stayed ahead of it and keep the site up and running pretty quick. That said, our dev team is stretched, we've accumulated some technical debt, and there's plenty of opportunity to optimize.

Without getting into specifics, we just found out that we'll be expecting a massive amount of traffic in the near future in a very short period time. On the order of several million hits in a few hours. Scaling is one thing, but this is several orders of magnitude greater than what we're seeing now.

We're a Rails app hosted on S3 using ELB, and Postgresql.

I wanted to field some recommendations for broad starting points for scaling and load testing given this situation.

  • Update: Sorry, EC2, late night :)
LastZactionHero
  • 269
  • 2
  • 10
  • Can you cache the information you'll need into static pages? S3 would serve those without a problem. Are you expecting a lot of action in your database? Reads? Writes? Are the queries specific to a single entity/user (not complex queries or massive joins)? There are a number of load testing applications out there that Google should help you find. Have you looked at EC2's auto scaling features? They have some that can trigger on CPU/memory, I believe. Also, static resources could be done with Cloudfront. – Steven Hood Apr 04 '13 at 04:40

2 Answers2

2

@LastZactionHero Pretty interesting question, let me answer you in detail, I hope you are talking about some e-commerce applications, enterprise or B2B apps doenst see spikes as such. Since you already mentioned that you are hosted your rails app on s3. Let me make couple of things clear. 1)You cant host an rails app on s3. S3 is simple storage service. Where you can only store files. 2) I guess you have hosted your rails app on AWS ec2 with a elastic load balancer attached above the ec2 instances which is pretty good.

3)You have a self managed Postgresql deployed on a ec2 instance.

If you are running on AWS you are half way safe and you can easily scale up and scale down.

I can see one problem in your present model, that your db. AWS has got db as a service. Thats called Relation database service.Which supports Mysql Oracle and MS SQL server.

RDS comes with lot of features like auto back up of your database, high IOPS etc.

But it doesnt support your Postgresql. You need to have or manage a self managed ec2 instance and run postgresql database, but make sure its fail safe and you do have proper back and restore system at place.

AWS provides auto scaling api and command line tools, pretty easy.

You dont have worry about the bandwidth issue etc, but I admit Angelo's answer too.

You can use elastic mem cache for caching your app. Use CDN if need to speed your app. RDS can manage upto 30000 IOPS, its a monster to it will do lot of work for you.

Feel free to ask me if you need any kind of help.

(Disclaimer: I am a senior devOps engineer working for an e-commerce company, use ruby on rails)

Jeevan Dongre
  • 4,627
  • 13
  • 67
  • 129
  • Thanks, this is helpful and largely confirms our direction. We're planning to switch to RDS- we have a handful of Postgresql-specific queries we need to sort out, nothing major, for the most part we just use ActiveRecord's methods, so I don't anticipate a huge problem. I'll look into Elastic Mem Cache as well. – LastZactionHero Apr 04 '13 at 14:02
1

Congratulations and I hope your expectation pans out!!

This is such a difficult question to comprehensively answer given the available information. For example, is your site heavy on db reads, writes or both (and is your sharding/replication strategy in line with your db strain)? Is bandwidth an issue, etc? Obvious points would focus on making sure you have access to the appropriate hardware and that your recipies for whatever you use to provision/deploy your hardware is up to date and good to go. You can often throw hardware at a sudden spike in traffic until you can get to the root of whatever bottlenecks you discover (and yes, you will discover them at inconvenient times!)

Regarding scaling your app, you should at least:

1) Cache whatever you can. Pay attention to cache expiration, etc. 2) Be sure your DB has appropriate indexes set up (essentially, you should have an index on any field you're searching on.) 3) Watch your logs closely to identify potential long queries, N+1 queries, long view renders, etc. 4) Do things like what Shopify outlines in this post: http://www.shopify.com/technology/7535298-what-does-your-webserver-do-when-a-user-hits-refresh#axzz2O0gJDotV 5) Set up a good monitoring system (Monit, God, etc) for each layer of your stack - sudden spikes in traffic can quickly bottleneck your application in unexpected places and lead to more issues. The cascade can happen quickly. 6) Set up cron to automate all those little tasks you currently do manually...that you will probably forget about doing once you're dealing with traffic spikes. 7) Google scaling rails and you'll see tons of good info. 8) etc, etc, etc...

You can use some profiling tools (rubyperf, or something like NewRelic, etc) Whatever response you get from them is probably best to be considered as a rough baseline at best. Simple reason being that your profiling is dependent on your hardware stack which will certainly change depending on actual traffic patterns. Pretty easy to do if you have a site with one page of static content...incredibly difficult to do if you have a CMS site with a growing db and growing traffic.

Good luck!!!

Angelo Chrysoulakis
  • 846
  • 2
  • 7
  • 21