Website Recommendations

Question

I would like advice in starting a new website that will be relatively high volume. Will need a database. I want a system that will grow with volume by adding hardware. What would be the recommendation for database, operating system, and hardware architecture ? Also, any ballpark cost estimates would be great. PLan to start out with a hosting service but may need a dedicated system from the start.

Answer 1

whatever technical solutions you're going to choose [ operating system, application layer, database engine ] - from very first day plan with scalability / high-availability in mind.

so - shard your data from day from day one, think how to spread load between servers generating pages.

it all really depends on what you're going to serve... couple of random addresses/books to take a look at [ linux/php/mysql biased ]:

• high performance websites + 2nd part
• jeremy zawodny's blog - guy used to work at yahoo, now works at crieglist
• mysqlperformanceblog
• countless presentations of google / you tube / facebook / wikipedia internals - just go go videos.google.com and search for clips about scalability longer then 10 min.

Answer 2

You might want to check this out. While not a direct answer to your question, and no ballpark costs on it, it features Jeff talking about the hardware and setup of this site. It was a bit of an eye opener for me.

http://www.runasradio.com/default.aspx?showNum=109

Answer 3

• Choose something that will scale horizontally vs. vertically. This means that would will be better suited for growth and uptime if you have many small web servers behind a load balancer vs. one big web server with a single point of failure.

• The OS and Database really depends on whoever is developing your web site. Seek counsel with them before making a decision. My default answer is Linux, Apache, Postgres. :D

• Consider hosting in the cloud (bluelock.com, gogrid.com) No huge capital costs, High Availability, fast scaling.

Answer 4

You'll definitely want to go with a dedicated hosting provider. Someone like RackSpace or Pair Networks comes to mind. I use RackSpace, and have been very happy with them for the past 4 years. When you call any provider, you should talk through what you're looking to do and they'll help you build the system.

Answer 5

I would consider dreamhost because they are initially cheap to host and you can switch over to their virtual, private hosting (4 gig of RAM, etc). It isn't technically a dedicated system, but it should come with many of the benefits.

Answer 6

Model your application and database. Using these models, determine where you need concurrency and where you need persistent storage. Use PostgreSQL for persistent storage and memcached for temporary storage. If you need long-term persistent storage for unstructured binary objects, use a cloud storage engine like Amazon S3 and a client library to interface with the API.

Develop and deploy your application on a cloud computing system like Amazon EC2. Define server roles and model the network architecture. Use Debian GNU/Linux as the base OS and an automated build system to bootstrap specific server roles.

Install statistical reporting tools like Munin and monitoring tools like Nagios when you have enough data to determine what "normal" is for your application.

When your application is live, monitor the growth of the data set and simulate a 1000 times increase of that set. Log your SQL and run some example queries through the PostgreSQL planner to determine where your database bottleneck is located. Trace that back to the application.

Determine a framework-dependent stats package for your application code. Review reports from this tool and determine what user actions are generating the slowest response times.