1

We're running a LAMP stack (Perl, MySQL) and have recently experienced roughly a tripling of our database load due to increased usage and shifting user behavior... which has overloaded our system.

Using a fairly standard 3-tier architecture, lots of webservers (doesn't seem to be the problem), a few specialty servers (don't think it's the problem) and a rather large MySQL database on modern Sun hardware with lota (72GB) of memory (the DB seems to be the current bottleneck).

Two questions I guess:

1) Anyone know of some resources they could point us to to help us work through our load issues? Buying hardware is definitely on the table; as is config changes...

2) Anyone out there with experience massively scaling a LAMP stack who would be interested in paid consulting work? We need someone immediately, preferably willing to travel to Raleigh NC to help on-site... First goal is to get out of the current crisis; second goal would be to help plan for another 10x usage increase over the next 2 years.

(As a sub-question to question #2: where would you go to look for such a consultant???)

Thanks in advance! -Steve

  • I don't have a particlar answer, but I do know that OmniTI http://omniti.com/ is good at scaling web applications. Have heard good things about them. – Dave Drager Sep 22 '09 at 17:46
  • Thanks for the responses! Very much appreciated. We are logging all requests and have identified a few badly optimized queries already which has helped; we'll definitely try for a thorough scrub of our SQL schema, possibly with some outside help. Thanks again! –  Sep 24 '09 at 15:19

2 Answers2

1

Buying hardware is definitely on the table; as is config changes...

Those would be #2 and #3 on my list of candidates, #1 being the database design. Usually, when the database load suddenly balloons without a corresponding growth in usage, there's a O(n^m) issue somewhere. A full table scan in a join, an inefficient denormalization, a n+1 glitch, maybe lock contention while inserting data, stuff like that. Solutions could be as simple as adding a few well-chosen indexes, or as complex as a redesign of the data model.

I haven't worked with MySQL in years, but what i remember is: you can have it log all requests and their execution times. You should look at that log and find out what exactly causes the database to become the bottleneck.

wallenborn
  • 257
  • 1
  • 2
0

The good news, if it is a DB issue, then many types of DB work can actually be done effectively via remote connection. If you're able to quickly set up a VPN into the server cluster, and a remote SSH / terminal account, then your scope of consultants broadens quickly.

I've never worked with Percona myself, but their MySQL scalability + performance book is really good, as is their work on Maatkit. Other consultants who give back to the community include Jeremy Cole @ Proven Scaling and the Open Query team.