The firm behind Healthcare.gov had top-notch credentials -- and it didn't help

CGI Federal, the lead contractor at Healthcare.gov, is a veritable black belt in software development. In 2012, it achieved the highest possible Capability Maturity Model Integration (CMMI) level for development certification, only the 10th company in the U.S. to do so.

CMMI offers process models to help an organization keep developments on track, ensure resources are in place, meet requirements, are measured, stay on budget and deliver value. But it does not offer technical approaches, such as how to conduct a test.

CMMI certifications frequently turn up as a requirement in government software development contracts. In fact, the Centers for Medicare & Medicaid Services, which oversaw the healthcare.gov project, says CMMI provides "the essential elements of an effective process."

In November, just a few weeks after Healthcare.gov launched, Henry Chao, the deputy CIO at the agency running the project, told Congress that 30% to 40% of the system had yet to be built. The government, clearly rattled by the problematic rollout, initiated a "tech surge" to bring in fresh help from Google, Red Hat and Oracle and help "development be more agile so Healthcare.gov can release improvements more rapidly."

Though CGI Federal got something of a black eye from the rollout, the CMMI certification it had did not come under fire , and no one has made a case that it should. Project requirements were changed late in the development cycle, warnings weren't heeded, and time for testing was cut short. Those actions are all anathema to CMMI's careful and measured development processes.

If a project that's based on CMMI runs into problems, process defenders will usually cite issues with management and decision-making. (That was certainly the case with Healthcare.gov.) But that's a default defense of CMMI generally. When a project fails, questions about CMMI may never come up. But when a project succeeds, CMMI may get the credit.

What good is CMMI?

So is CMMI of value? The private sector has clearly mixed views.

Joel Basgall the CEO of custom software developer firm Geneca, says he has never had a client who uses CMMI. (Geneca helped build healthcare exchanges for private companies.) "When we're out in marketplace, we compete more on what we can do, and how we do it, as opposed to the fact that we have a process," Basgall said.

He argued that the government should pick developers based on who has the best chance of success and drop CMMI certifications as a contract requirement.

"Everybody they (the government) go to will have a process -- nobody can function without one," said Basgall. "CMMI isn't actually measuring how good the process is, it's measuring whether it's defined, and if it's managed, and optimizing," he said.

CMMI arose some 25 years ago via the backing of the Department of Defense and the Software Engineering Institute at Carnegie Mellon University. It operated as a federally funded research and development center until a year ago, when CMMI's product responsibility was shifted to a private, profit-making LLC, the CMMI Institute LLC. The Institute is now owned by Carnegie Mellon.

Getting CMMI certification is not simple or cheap. It has five levels of process maturity, each taking about two years to complete. Going through these processes can cost a company thousands of dollars and may even require dedicated personnel. Certifications require appraisals by independent experts, and periodic re-appraisals.

Given that the CMMI Institute is now a self-supporting firm, any requirement that companies be certified by it -- and spend the money needed to do so -- raises a natural question.

CMMI's new status

"Why is the government mandating that you support a for-profit company?" said Henry Friedman, the CEO of IR Technologies, a company that develops logistics defense related software and uses CMMI.

"There is the incentive for them to grow, and the way you grow is you either sell your product or services to more customers, or increase the number of products and services that you sell to the same customer," said Friedman.

Friedman is concerned that government managers will escalate CMMI requirements as a result of increased commercialization. This can be an issue. For instance, he was recently involved in a bid that required CMMI certification, with the military agency wanting certifications in three areas: development, services and acquisitions. At the time, he was aware of only two organizations in the world that had all three certifications, effectively shutting out a number of would-be bidders for the project.

The requirement for three was dropped once vendors complained, he said.

The scenario described by Friedman shows how CMMI certification requirements could limit bidders for government contracts, exacerbating a problem former federal CIO Vivek Kundra complained about in 2011, just before he left his post. He said government IT contracting was almost a cartel, comprising a handful of contractors that benefit from government spending "because they understand the procurement process better than anyone else."

The case for CMMI

CMMI supporters, however, say it would not have survived if it did not add value.

Kirk Botula, CEO of the CMMI Institute, said CMMI has gone far beyond its role in government procurement, is now used in the private sector and has a strong following overseas -- especially in China and India.

"You can learn through trial and error, which is how most folks do it," said Botula, of software development, "or you can benefit from best practice, from proven approaches and use it as a roadmap to align your business goals to your operational capability. (CMMI) is a consistent way of doing that."

Last summer, the CMMI Institute conducted its first "customer satisfaction" survey of users worldwide. Nearly 1,900 responded with generally high marks. On a scale of one to 10, fully half of the respondents rated CMMI a 9 or 10 when it comes to evaluating organizational performance; 41% gave it a 7 or 8; and only 9% rated it at 6 or lower.

The Institute provided the raw survey results with comments from responders pointing to improvements as a result CMMI. One U.S. customer wrote: "Most organizations would benefit from the model, but many would lack the very large commitment it takes to follow it through."

Hillel Glazer, the CEO of consulting firm Entinex and author of the CMMI FAQ, a book-length analysis of the system, rejects the idea that the CMMI Institute will raise barriers or foist new requirements on the market. By being independent of the university, Glazer said, the Institute can better focus on the needs of the market.

"Moving towards a 'for-profit,' operation can only influence the market on its own footing," said Glazer.

Jim Johnson, the founder the Standish Group, which has a database of some 50,000 development projects, said "the problem with processes and tools is they leave out leadership, they try to create a cookbook that people can follow, but eliminate the leadership part of it."

Johnson is on the fence about CMMI, but said it "would be down on the long list of things that I would do."

Having the right leadership, environment, executive sponsors, and agile process are among the things that Johnson said he would make sure are in place before looking at CMMI.

And as far as trial-and-error goes, Johnson points to the work needed to create the first light bulb and Apple's iPod.

"Sometimes trial and error is good," he said.

Patrick Thibodeau covers cloud computing and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at @DCgov or subscribe to Patrick's RSS feed. His e-mail address is pthibodeau@computerworld.com.

See more by Patrick Thibodeau on Computerworld.com.

Read more about enterprise architecture in Computerworld's Enterprise Architecture Topic Center.