However Bingham had a tough time during the Iowa caucuses in early January 2008 due to the site’s increased use of memcache. A cronograph job designed to serve front page stories and recommended diary entries to anonymous users meant the site was brought to its knees every five minutes as its Web server caches were simultaneously updated.
A weekend spent reading K&R and the removal of a few site features ensured the problem was fixed before the New Hampshire primary a few days later, and a hardware upgrade stopped most problems by the Pennsylvania primary in April.
Along with server upgrades and consolidation, Bingham’s time was spent improving back-end code and hunting down inefficiencies in Perl code and database tables and queries. He said the server upgrades improved the function of the already optimised code and reduced maintenance and administration requirements.
The rise of the Kos
The site has come a long way since Moulitsas ran it using the weblog management system Moveable Type on a shared server. Bingham shifted the site onto Scoop running on a dual Pentium 4 with 1GB of RAM. As registered users increased, the back-end was upgraded to a server cluster with a couple of dual Xeon Web front-end servers and an Opteron database server for load distribution.
A fleet of 10 dual Xeons each with 2GB of RAM ran alongside the then two dual Opterons with 8GB of RAM and a dual Xeon each for image serving and memcache. A Xen virtual machine ran MySQL backups while search and outbound mail was delegated to the since humbled dual Pentium 4.
Daily Kos now boasts six quad core Xeons with 8GB of RAM that network boot from a central NFS root, with local scratch disks for logs and other machine specific files, and each running 1GB of memcache shared with a total memcache pool of 6GBs.
“With the network file system root and network booting for our Web servers, it's easy to add new servers to the Web server pool on a moment's notice,” Bingham said.
The database now runs on two eight core Xeons with 16GB of RAM each, and dedicated drives for the operating system and locally mounting /tmp. MySQL runs on six 73GB disks in a RAID-10 configuration, with the two servers are in a master/slave setup, while an additional quad Xeon works with the dual Pentium 4s for outgoing mail and search and acts as the origin point for the content delivery network. A Xen virtual machine runs MySQL enslaved to the master and stores back-up snapshots of the database.
The greatest performance improvements to site stability, Bingham said, was the move from an Apache front-end proxy to a lighttpd with a memcache capable mod_magnet. A custom mod_mcpage handled caching pages for anonymous users, and some heavily requested files on the Web servers were served from temporary file system mounts.