On Designing and Deploying Internet-Scale Services

Very interesting paper by James Hamilton. The paper gives a set of best practices for designing and developing operations-friendly services.

The system-to-administrator ratio is commonly used as a rough metric to understand adminis-
trative costs in high-scale services. With smaller, less automated services this ratio can be as low as
2:1, whereas on industry leading, highly automated services, we’ve seen ratios as high as 2,500:1.
Within Microsoft services, Autopilot [1] is often cited as the magic behind the success of the Win-
dows Live Search team in achieving high system-to-administrator ratios. While auto-administration
is important, the most important factor is actually the service itself. Is the service efficient to auto-
mate? Is it what we refer to more generally as operations-friendly? Services that are operations-
friendly require little human intervention, and both detect and recover from all but the most obscure
failures without administrative intervention. This paper summarizes the best practices accumulated
over many years in scaling some of the largest services at MSN and Windows Live.