Joe Latrell, IT manager and lead programmer for GetMyHomesValue.com, a real estate data services company in Lancaster, Pa., knows that it's all too easy for even a knowledgeable and experienced IT veteran to make mistakes while managing a complex server-consolidation project. "You have to think about everything," he says. "It can be a minefield."
Server virtualization projects are usually easy to justify on both financial and operational grounds, but that doesn't make them foolproof to execute. Pitfalls, such as inadequate planning, faulty assumptions or failure to quickly detect postdeployment glitches, can entrap consolidation project leaders and team members at almost every stage.
"Every time [we] felt that we covered every base, that every single thing had been looked at ... that's when the danger started," says Latrell, whose project experienced a variety of woes, including underpowered servers, configuration snafus and budget constraints.
Avoiding disaster while keeping a complicated consolidation project on schedule and within budget isn't easy. In fact, Latrell believes that making at least a few mistakes along the way is inevitable. "It will go wrong: Be prepared," he warns. "On the other hand, planning and learning from others will keep you from making the big and obvious mistakes."
Plan for success
While even the most thorough, painstaking planning can't completely eliminate project mistakes, building a detailed virtualization design and deployment strategy will help minimize the number of gotchas. "Planning is really key for server consolidation," says Justin Gallagher, senior IT consultant at KDSA Consulting LLC in North Andover, Mass.
Five steps to successful server consolidation planning
1. Begin by researching. Having solid understanding of the different consolidation techniques and technologies will help you pinpoint the approach that best meets your needs.
2. Set operational and financial goals. Identify exactly what you hope to accomplish in terms of scope, performance and costs before you begin designing the system.
3. Create a schedule. To minimize disruption to everyday business and IT operations, set a timeline with defined benchmarks. Be sure to build in some extra time to accommodate inevitable project setbacks.
4. Put it in writing. A document describing project goals, system design details, integration specifics, server management responsibilities and other key points will help you, your team and external partners stay on track.
5. Build support. As with any major IT project, getting buy-in from stakeholders is essential for a successful consolidation initiative. Make sure your business users understand the benefits, and any pitfalls, of virtualization.
Thorough planning creates a road map that helps managers gather the knowledge required to avoid most major problems. "I think people aren't spending enough time thinking about the issues of the existing workloads and how you migrate those into a virtual environment, and what does that mean in terms of , cost structure, ongoing expense and high availability," says Jeff Nessen, IT consolidation practice manager at Logicalis, a systems integrator in Bloomfield Hills, Mich.
Gallagher says that consolidation planning also needs to address an organization's future needs. "Look at what you're going to do a year, three years and five years from now," he suggests. Gallagher notes that servers, software and other system elements need to be planned with an eye toward anticipated growth. "You don't want to get yourself in a situation where you do this whole big upgrade and then you find you need more [server capacity] later on."
Jason Cooper, a consultant at C/D/H Technology Consultants in Grand Rapids, Mich., agrees that every consolidation plan needs to address scalability. "From the standpoint of server virtualization, it's very important to have a system that scales and meets the performance need of the load you're putting on it," he says. "We often run into issues with organizations that either didn't allocate enough storage, or simply didn't correctly anticipate the amount of server power that was going to be needed to facilitate their server consolidation project."
It's extremely common to overestimate the physical-to-virtual consolidation ratio, experts say.
Planning is particularly critical when managing a data center with outmoded equipment and a limited budget. The moment he arrived on the job, Latrell inherited a motley collection of servers, including converted desktop PCs and a mix of underpowered stand-alone and rack-mounted machines. He was determined, however, to streamline the collection into a uniform line of rack-mounted servers and, in the process, to winnow down the total number of units from 23 to 12.
"We understood we didn't have the budget to just go out and buy 12 new servers, so we decided to purchase the machines as we could afford them," he says. "We planned in advance to implement one piece at a time, and it's worked out well."
Understand the technology
Running headlong into consolidation without fully understanding the technology involved, and its requirements, is a good way to doom the project from the very start. "Most mistakes I've seen are made when someone goes out and buys a couple copies of virtualization software, implements two servers and then just starts migrating things onto them -- I call it willy-nilly deployment," says Nessen. "They're stuck with an environment that's less than optimal for migrating their data center into."
Nessen says that the key to an optimal virtualized environment is component compatibility and the use of widely recognized standards. "The biggest success is when you standardize your hardware platform and your software environment as much as you can -- the same hypervisors, the same underlying hardware and all those pieces," he explains.
Harvey R. Morris, president of BL&S Technologies LLC, says his company is currently working to consolidate 14 physical servers to just four or five. The IT consultancy's initial strategy called for an existing server to back up a new virtualization server. But that turned out to be impossible. "The [old] server was not 64-bit-compatible, which can be a problem if you're running a 64-bit operating system on the new server," Morris says. Fortunately, the mistake was caught in time and a different, compatible system was used.
Eric Mynster, IT operations manager at Mercy Memorial Hospital System in Monroe, Mich., says he was able to gather insight from colleagues at several other area health care facilities. "We talked to three or four hospitals ahead of time and learned some important lessons from them," he says. The two biggest tips he got: Use virtualization migration software, and use portable storage technology to quickly and conveniently move data from remote servers to the organization's on-site systems.
Mynster feels he was lucky, since his project began just a couple of months after the other hospitals had completed theirs. "So we already knew the pitfalls, and we wrote our plan around that," he says.
Failing to synchronize new plans with ongoing system life-cycle and business needs is guaranteed to mess up almost any IT project, but server consolidation requires even more attention to ongoing events, since servers lie at the heart of virtually all business-critical tasks.
Morris says he was careful to coordinate his initiative with his server replenishment timetable. "We had actually looked at the virtualization project about 12 months ago, but we decided to hold off until we were ready for the next server replacement cycle," he says. Morris also wanted pick the right time of year to begin the transition.
The accounting firm didn't want the changeover, or any systems work at all, to occur during tax season. "We have quite a time from January through April when we wouldn't consider making these types of changes," Morris says.
Mynster, meanwhile, worked to ensure that older servers, which were expected to perform flawlessly in a new, virtualized data center, would be up to the job. "About 50 of the servers were anywhere from three to 10 years old," he says. "Anytime you're talking about moving hardware that's been in place for 10, nine or eight years, you get a little nervous."
Before the changeover, half of Mercy's 200 servers were located on-site, while the other half were based at a third-party vendor located more than an hour away from the hospital. Mynster turned to Novell's PlateSpin Migrate to convert and transfer data stored on the 100-plus servers located at its hosted sever vendor to its in-house data center. A consolidation ratio of 18:1 allowed the on-site servers to absorb the extra load with no problems.
The software allowed Mercy's systems integrator, C/D/H, to effectively virtualize the servers ahead of time. The firm captured the remote server images onto USB storage, ran them through a synchronization process and brought the new machines live before the off-site servers were disconnected. The approach gave Mynster and his team the time they needed to check for any lurking operational glitches.
The process "allowed us to do a very fast migration," Mynster said. The work -- from finalization of plans to implementation -- took just under three months.
Watch for warning signs afterward
Not paying attention to error logs and other system-generated clues is perhaps the biggest postdeployment mistake made in consolidation efforts. Latrell recalls an incident at his shop.
"We have lots of little programs that send e-mails out; some of those programs stay asleep for months at a time," he explains. One of those routines wasn't correctly updated during the virtualization conversion. "Somebody had used the domain name for the name of the server, and we didn't see it," Latrell says. One day, the routine woke up and began sending e-mail that was undeliverable. "It showed up in the error log," Latrell says.
The problem was an easy fix. "We went into the code, found the problem, pointed that particular set of code to the new server and then located all the mail that went out from that particular process and resent it manually."
With fewer servers in play, many enterprises make the mistake of cutting back on support and backup technologies when they should actually be reinforcing their safety net. "When you've got 10 machines running on a single physical machine, that power supply really needs to be at a good level and your cooling needs to be right," says Steven Meek, president of The Fulcrum Group Inc., a systems integrator in Keller, Texas. "There are a lot of foundational things that need to be in place before you consolidate your servers."
Finally, even as he worked to avoid mistakes, Latrell says, he kept one basic thought in mind. "If there's trouble, I'm the one the boss talks to," he says. "The buck stops here."
John Edwards is a technology writer in the Phoenix area. He can be reached at jedwards@gojohnedwards.com.
