How the Kossacks conquered America

Fresh-faced US President Barack Obama is, like his campaign maxim, an emblem of change. The traditional vehicles for the dissemination of information and influence on the public — the mass media and old-style campaign letter-boxing — changed in the years leading up to the 2008 election, as political Web sites received unprecedented popularity from wider audiences.

Daily Kos, a grassroots Democratic Web site, and its Kossacks (as the members happily call themselves), has come to symbolise the new power of political blogs, evidenced by the influential US politicians, including former president Jimmy Carter and Senator Harry Reid, who drop by to connect with its burgeoning list of activists and potential campaign volunteers.

The Internet has fundamentally changed the way political messages are disseminated, Daily Kos founder Markos Moulitsas told MSNBC.

“Technology absolutely has an impact [on politics]; it has allowed the Obama campaign to speak directly to its supporters without the need to go through media filters,” Moulitsas said.

“When John McCain selected Sarah Palin as a running mate, Daily Kos community ran into action, and [disclosures about] her membership in the Alaskan Independence Party stemmed from this open source grassroots reporting that occurs on Daily Kos and other sites like it.”

As the site's burgeoning popularity thrust Moulitsas into the US political spotlight, including multiple appearances on major television networks, Daily Kos chief technical officer Jeremy Bingham had to deal with soaring traffic loads and fluctuating peaks. Daily Kos grew from a humble blogging site in 2002 running on a single shared server to a formidable hub for liberal activism with more than 180,000 registered users.

Bingham spends most of his time checking code and server status, responding to e-mails and researching ideas to improve the open source architecture of the Web site. “It's certain that someone will need help with something,” he told Computerworld.

The lead-up to the 2008 presidential elections saw big improvements to the site's back-end code, which ironed out inefficiencies and reduced server load.

“The site didn't hold up very well at all in 2004; it crashed constantly, and I had to scramble around a lot trying anything I could to keep the site up,” Bingham said. “Still, I was able to learn from that experience and … was able to keep the site humming on [congressional election day in] 2006.”

“By 2008, I was able to take the lessons learned, along with improvements made during 2007, and get the site tuned up to the point that it ran without a hitch on Election Day,” he said.

Daily Kos served up more than nine million pages on the day of the US general election, amazingly without a hitch, although Bingham concedes there were a few errors with its election map.

Page Break

However Bingham had a tough time during the Iowa caucuses in early January 2008 due to the site’s increased use of memcache. A cronograph job designed to serve front page stories and recommended diary entries to anonymous users meant the site was brought to its knees every five minutes as its Web server caches were simultaneously updated.

A weekend spent reading K&R and the removal of a few site features ensured the problem was fixed before the New Hampshire primary a few days later, and a hardware upgrade stopped most problems by the Pennsylvania primary in April.

Along with server upgrades and consolidation, Bingham’s time was spent improving back-end code and hunting down inefficiencies in Perl code and database tables and queries. He said the server upgrades improved the function of the already optimised code and reduced maintenance and administration requirements.

The rise of the Kos

The site has come a long way since Moulitsas ran it using the weblog management system Moveable Type on a shared server. Bingham shifted the site onto Scoop running on a dual Pentium 4 with 1GB of RAM. As registered users increased, the back-end was upgraded to a server cluster with a couple of dual Xeon Web front-end servers and an Opteron database server for load distribution.

A fleet of 10 dual Xeons each with 2GB of RAM ran alongside the then two dual Opterons with 8GB of RAM and a dual Xeon each for image serving and memcache. A Xen virtual machine ran MySQL backups while search and outbound mail was delegated to the since humbled dual Pentium 4.

Daily Kos now boasts six quad core Xeons with 8GB of RAM that network boot from a central NFS root, with local scratch disks for logs and other machine specific files, and each running 1GB of memcache shared with a total memcache pool of 6GBs.

“With the network file system root and network booting for our Web servers, it's easy to add new servers to the Web server pool on a moment's notice,” Bingham said.

The database now runs on two eight core Xeons with 16GB of RAM each, and dedicated drives for the operating system and locally mounting /tmp. MySQL runs on six 73GB disks in a RAID-10 configuration, with the two servers are in a master/slave setup, while an additional quad Xeon works with the dual Pentium 4s for outgoing mail and search and acts as the origin point for the content delivery network. A Xen virtual machine runs MySQL enslaved to the master and stores back-up snapshots of the database.

The greatest performance improvements to site stability, Bingham said, was the move from an Apache front-end proxy to a lighttpd with a memcache capable mod_magnet. A custom mod_mcpage handled caching pages for anonymous users, and some heavily requested files on the Web servers were served from temporary file system mounts.

Page Break

“Using a content delivery network to serve up our images has also worked pretty well, with faster load times for images,” Bingham said.

Bingham said he will ignore cloud computing and instead ramp-up the site's dedicated hosting service with provider Voxel.net. He said the service upgrade will allow capacity to be scaled up and down in response to traffic spikes and to provide fine tuning for the servers.

Democrats open to change

The philanthropic ideals inherent in open source chime well with the Kossacks' liberal values, so it is no surprise that proprietary technology is a scarcity on the site. Moreover, Bingham is giving back to the community by freely publishing modified code that has improved the efficiency of Daily Kos back-end systems.

Bingham said open source is not only cheaper and more flexible than commercial alternatives, but it effectively transforms the developer community into an enormous support group.

“[Open source] works well, and I like the freedom that it gives the user,” he said. “For instance, if we need a new feature added to Scoop, or find that something is running too slow, I can just go in and fix it, rather than wait for the vendor to release a new version or a bugfix.”

IT-savvy Kossacks help Bingham to keep the site bug-free, and offer some invaluable advice on open source cosmetics.

“Sometimes they write in with an idea I'd never thought of, or a suggestion from their line of experience that can be very handy. As far as keeping bugs off the site, they're the best people to report them,” Bingham said. “If something breaks on the site, they're not shy about letting me know about it.”

Bingham, a Scoop developer from his prior role as a consultant, returns his polished code into the community. He released mod_mcpage, a module to cache pages in lighttpd, which helps reduce the work load on his Apache web server back ends, and is adding more improvements to make it more useful to other developers.

He said open source is gaining traction in corporate and government IT shops, but concedes advocates must press its advantages to management for adoption to increase significantly.