Serving small static files: which server to use ?

Update 1 (Mar 29, 2011): Logging and compression now disabled for all servers
Update 2 (Apr 14, 2011): Added results for Nginx 1.0

Introduction

The goal of this benchmark is to compare several web server or caching server regarding their respective ability and performance to server small static files. The servers are optimized, contrary to my previous benchmark, where each server was tested using its default settings. I consider in this benchmark only the most performant open source servers, namely Varnish Cache, Nginx, Lighttpd, Apache Traffic Server as well as G-WAN (free, but not open source), as it was the clear winner of the previous test.

Continue reading

Serving static files: a comparison between Apache, Nginx, Varnish and G-WAN

Update 1 (Mar 16, 2011): Apache MPM-Event benchmark added
Update 2 (Mar 16, 2011): Second run of Varnish benchmark added
Update 3 (Mar 16, 2011): Cherokee benchmark added
Update 4 (Mar 25, 2011): New benchmark with the optimized settings is available

Introduction

Apache is the de facto web server on Unix system. Nginx is nowadays a popular and performant web server for serving static files (i.e. static html pages, CSS files, Javascript files, pictures, …). On the other hand, Varnish Cache is increasingly used to make websites “fly” by caching static content in memory. Recently, I came across a new application server called G-WAN. I’m only interested here in serving static content, even if G-WAN is also able to serve dynamic content, using ANSI C scripting. Finally, I also included Cherokee in the benchmark.

Continue reading

Convert WMA to MP3

Here is a simple script I’m using to convert wma music files to mp3:

#!/bin/bash

current_directory=$( pwd )

#remove spaces
for i in *.wma; do mv "$i" `echo $i | tr ' ' '_'`; done

#remove uppercase
for i in *.[Ww][Mm][Aa]; do mv "$i" `echo $i | tr '[A-Z]' '[a-z]'`; done

#Rip with Mplayer / encode with LAME
for i in *.wma ; do mplayer -vo null -vc dummy -af resample=44100 -ao pcm:waveheader $i && lame -m s audiodump.wav -o $i; done

#convert file names
for i in *.wma; do mv "$i" "`basename "$i" .wma`.mp3"; done

rm audiodump.wav

On Designing and Deploying Internet-Scale Services

Very interesting paper by James Hamilton. The paper gives a set of best practices for designing and developing operations-friendly services.

ABSTRACT
The system-to-administrator ratio is commonly used as a rough metric to understand adminis-
trative costs in high-scale services. With smaller, less automated services this ratio can be as low as
2:1, whereas on industry leading, highly automated services, we’ve seen ratios as high as 2,500:1.
Within Microsoft services, Autopilot [1] is often cited as the magic behind the success of the Win-
dows Live Search team in achieving high system-to-administrator ratios. While auto-administration
is important, the most important factor is actually the service itself. Is the service efficient to auto-
mate? Is it what we refer to more generally as operations-friendly? Services that are operations-
friendly require little human intervention, and both detect and recover from all but the most obscure
failures without administrative intervention. This paper summarizes the best practices accumulated
over many years in scaling some of the largest services at MSN and Windows Live.

Another real-time search and indexing system built on Apache Lucene.

From their website:

Zoie is a mature open source project and has been deployed in a real-time large-scale consumer website: LinkedIn.com handling millions of searches as well as hundreds of thousands of updates daily.

All Zoie releases have gone through extensive functional and performance testing by LinkedIn before made public. All major versions are released after a trial period on the production environment.

In a real-time search/indexing system, a document is made available as soon as it is added to the index. This functionality is especially important to time-sensitive information such as news, job openings, tweets etc.