Virtualizing Oracle software: Don't pay for what you don't need

Virtualization and private clouds have delivered hardware savings in a big way for most enterprises. Software savings, however, are proving more elusive, and that's been particularly true when it comes to virtualizing Oracle databases, according to several sources.

While virtualization has enabled server consolidation ratios of 3:1 or more, businesses may see little or no reduction in associated software costs. In some cases enterprise software licensing can be so expensive that it overshadows the cost of the rest of the system stack combined, says David Welch, chief technology officer at House of Brick Technologies, an integrator with expertise in Oracle software licensing issues.

(Editor's note: During the months of reporting this story, we contacted Oracle 13 times, and the vendor's spokespeople declined to comment on this story on three separate occasions.)

Some consultants and a handful of IT executives say they face licensing obstacles with many enterprise software vendors, but Oracle agreements can present some of the most confusing compliance issues, especially for customers that fall under Oracle's processor-based licensing models. "Using Oracle on VMware provides no licensing savings, just operational cost savings," says R Wang, principal analyst and CEO at Constellation Research.

"Virtualization is an area where customers get creamed all the time," says Craig Guarente, CEO of software auditing and compliance consultancy Palisade Compliance. Both Guarente and Welch say that, all too often, organizations end up overpaying for new Oracle licenses as well as software maintenance after consolidating servers or adding new licenses in a virtual environment.

Several IT executives contacted for this story declined to speak on the record. One executive, "Tom," has dealt with Oracle as the CIO for several Fortune 500 corporations, and agreed to speak only on condition that his name and company affiliation not be used. "Part of the challenge is knowing what others are paying," he says, and that information is considered confidential. But professional negotiators can be helpful because they know what the average company is paying for any given product, he adds.

Some Oracle customers are "p.o.'d" about virtualization, Guarente says.

Story continues on next page.

Negotiating with Oracle: Experts' tips

• When executing an Oracle software license agreement, visit the Oracle contracts page and print out or store copies of all online policy documents to which the contract refers as a class.
• Be sure to include the Oracle Partitioning Policy document (PDF), Oracle Processor Core Factor Table (PDF), Software Investment Guide, Licensing Data Recovery Environments (PDF) and Oracle Software Technical Support Policies (PDF). These have information not otherwise specified in the customer's Oracle Software Licensing Agreement (OSLA).
• The documents can change over time, so it is important to have copies of the policies that are in force at the time the OLSA is executed.
• For more information on Oracle licensing in a VMware environment read the VMware white paper, "Understanding Oracle Certification, Support and Licensing for VMware Environments" (PDF).
• Avoid overspending on new Oracle licenses when moving servers to a virtual environment, especially when it comes to vSphere clusters. Oracle may ask users to pay a licensing fee for every server in a cluster -- not just the servers where the Oracle products are installed and/or running -- "but that condition is not spelled out in the software license or associated policy documents," says Guarente.
• Inventory your Oracle licensing paperwork before starting a server consolidation project to see if you can allow maintenance contracts to lapse selectively for Oracle licenses associated with retired physical servers. Note that Oracle's contracts group individual licenses within so-called license sets. Oracle's matching support levels policy states that you can't cancel maintenance and support on a subset of licenses within a license set without also cancelling those licenses. That may trigger a re-pricing for the rest of the agreement. So renegotiating the contract may not save you any money.
• Carefully consider the implications of any invitation to migrate all of your Oracle licenses into a single agreement. Such initiatives rarely work in the customer's favor, professional negotiators say. Keep Oracle software licensing contracts separate whenever possible for maximum flexibility. Also keep in mind that Oracle's License Sets policy works against allowing maintenance and support to lapse on all product licenses within a given Customer Service Identifier (CSI).
• According to what's commonly known as Oracle's 10-day rule, you're entitled to leave one node unlicensed in a vSphere cluster -- even if you dedicate every other physical server in the cluster to Oracle -- for up to 10 cumulative days, so long as that node is named as a designated failover node. The failover node must also share a storage array with the source node. See Oracle's Software Investment Guide and Licensing Data Recovery Environments documents for details.

Sources: House of Brick, Palisade Compliance

-- Robert L. Mitchell

One client called Palisade after installing Oracle database software on two servers in an eight-server vSphere cluster. Oracle discovered the configuration during a software audit and demanded that the customer buy a license for every processor and server in the cluster.

We haven't noticed other enterprise software vendors that assert the full-cluster licensing claim. David Welch, chief technology officer, House of Brick Technologies

That's unusual, says Welch. "We haven't noticed other enterprise software vendors that assert the full-cluster licensing claim," he says. Oracle's per-processor licensing fees start at $47,500, so charges can add up quickly. To make matters worse, the customer had also unknowingly installed the more expensive Enterprise Edition of the Oracle database instead of the Standard Edition.

Before the audit, the company had been paying Oracle $50,000 per year. The list price for licensing the new configuration: Just over $1 million. The customer ended up reconfiguring the vSphere infrastructure to limit the number of servers in the cluster, and bought more Oracle licenses. Guarente wouldn't say exactly how much the customer saved, but did say the customer paid a fraction of that original $1 million bill.

The company could have achieved the same goal without paying anything additional had it reconfigured the cluster to run Standard Edition before the audit took place. Or it could have gone back to Oracle and bought more licenses, Guarente says. "The first option would have resulted in no additional fees to Oracle. The second would have resulted in some fees, but far less than what they ended up paying."

Bob Clarke, manager of enterprise services for the Office of Technology in the State of Indiana, is considering his options for saving money. Like the customer described above, he also faces the requirement to license every core and server in the cluster.

"It gets very pricey," he says, but he hopes to make the numbers work by consolidating some of the existing licenses for Oracle software used by separate departments. "We will create a vSphere cluster that's dedicated to Oracle. We will comply with their licensing by doing it that way," he says.

But user experiences vary widely. "Bill," a CIO at another Fortune 500 company who also asked not to be named, says he has never been asked to pay to license every core in a server used in a virtual environment when not all are running Oracle, nor has he been asked to buy Oracle licenses for every server in a vSphere or any other virtual server cluster.

The key is in the negotiations and the customer's relationship with Oracle. "It's battle. You have to work hard," he says, but he's always been able to negotiate a satisfactory contract. Then again, he admits, he's never been through an audit.

Being asked to limit vSphere clusters just to Oracle isn't always efficient, especially for large data centers, and it's not necessary, professional negotiators say. "There is nothing in the contracts that say you have to pay for all the servers in a cluster," Guarente says. Although the software vendor has published separate policy documents that attempt to address this issue, Welch says there's nothing in black and white to back up the assertion that every server in a vSphere cluster must be licensed if Oracle software is installed or running on any one of those servers.

"The biggest issue for me is that these documents are not part of the contract the customer signed," says Guarente. "Oracle has very interesting policies around virtualization, and none of those policies are located in the contracts."

The problem with soft partitioning

One of the most relevant policy documents on the matter, the Oracle Partitioning Policy (PDF), distinguishes between hard partitioning and soft partitioning technologies with respect to licensing. With hard partitioning, customers can limit the number of processor licenses needed to those running Oracle. With soft partitioning, they cannot.

Oracle defines Microsoft Hyper-V and VMware as soft partitioning, although many technologists will argue that the features and methods used in VMware are no different than those that appear in Oracle's list of hard partitioning technologies, Guarente says. "Given that this is a competitor's offering, Oracle will always consider this as a form of soft partitioning," he explains.

The one exception, he adds, is under Microsoft's Windows Azure cloud offering, which uses Hyper-V. In that case, Oracle can be deployed within a virtual machine and the customer pays only for the resources used. "The reliance is on the provider to control the size of the virtual machines," he says.

Even if the customer's software license refers to the Partitioning Policy, that document includes a note stating that the policy "is for informational purposes only," and "does not constitute a contract or a commitment to any specific terms."

So does the policy apply or not?

Guarente says when he challenged one contract based on published policy material for an organization Oracle was auditing, he received that very same response from License Management Services, Oracle's auditing group. "They came back to us and said the documents were for educational purposes only and not binding. Fascinating." That business was able to successfully defend its position with Oracle, he adds.

"It's the art of working with Oracle versus the science of compliance. It's all very muddy," Guarente says. Furthermore, the list of approved hard partitioning technologies typically does not appear within the list of documents referenced in the Oracle Software Licensing Agreement (OSLA) that customers sign, says David Blake, CEO at UpperEdge, an IT sourcing consultancy.

Nonetheless, if one follows the Oracle Partitioning Policy, every processor and core within a single physical server must be licensed for Oracle in a VMware environment, even if the Oracle software is installed or running on just one core. House of Brick's Welch tells his clients to pay to license every processor and core in that situation. "We tell everyone that Oracle is right. That policy has been there since 2002," he says.

The subcluster debate

Running Oracle on a dedicated group of servers within a vSphere cluster, sometimes referred to as a subcluster, is a different matter, however. "Oracle likes to tell prospects and customers that if they are running Oracle on any physical server within a large vSphere cluster they have to license the entire cluster for Oracle. Nothing could be further from the truth," Welch says.

"The contract states that you must license any physical server on which you have installed or are running Oracle binaries. But you don't have to license other servers in that cluster," he insists. "Amazingly, many organizations don't know that."

Welch says he has negotiated on this point many times for enterprise clients. "When we have asked Oracle auditors if they could show us where it says in writing that every server in a vSphere cluster must be licensed for Oracle they said it was an unwritten policy."

Blake concurs. "Overall, the policy does not specifically call out the cluster scenario, so we advise clients to challenge Oracle" when a request is made to license every server in a vSphere cluster, he says.

In some cases, Welch adds, Oracle has asked some customers to license failover nodes in the cluster as well, but you're entitled to leave one node unlicensed, even when every physical server in a vSphere cluster is dedicated to Oracle, he says. Oracle's Software Investment Guide includes what's commonly known as the 10-day rule. "It says that you can run Oracle on an unlicensed physical server for up to 10 cumulative days per year as long as it's a named, designated node, and you fail back."

So why is this happening? Oracle is such a large organization that "people just make the assumption that there's no way that an Oracle account representative could be representing something to an organization that would not be contractual or binding," Welch says.

But he cautions, "Corporations need to examine their licensing agreements rather than just listening to Oracle. And that can include case law searches should a licensee care to check if the largest relational database vendor in the world has ever made any legal filings on this topic."

As to the issue of licensing failover nodes, Welch elaborates: "VMware High Availability accomplishes the same thing as its active/passive HA predecessors such as IBM HACMP, HP Service Guard or Veritas Cluster Server. We never saw Oracle assert [that] the passive node had to be licensed in those legacy clustering technologies. But suddenly it is making a fuss about vSphere clusters," Welch says.

One potential reason for the discrepancy, Welch asserts, could be that those legacy HA failover technologies don't present a threat to Oracle's processor-based license revenue in the way VMware does by enabling massive server consolidations.

Overspending on Oracle

The costs of over-licensing can be substantial, and Welch thinks many customers make unnecessary "donations" to Oracle when it comes to virtualization. "When buying licenses for a new project on a vSphere cluster, we see 150% to 300% overspend on Oracle licenses alone," he says. "This overspend dynamic with respect to VMware is unique to Oracle, in our observation," Welch says.

That's not to say that other enterprise software doesn't present challenges. For example, Microsoft's licensing policies for SQL Server on VMware took a significant turn for the worse with SQL Server 2012. But in that case the heavier licensing requirement was "unambiguously contractual," he says.

One House of Brick client had hundreds of vSphere hosts, only wanted to run Oracle on a few of them and didn't want to divide up the cluster to dedicate one to Oracle. "For that corporation, a national leader in their industry, the ramifications would have scaled into the tens of millions of dollars of unnecessary additional spend with Oracle," Welch says. Executives there are still trying to decide what course to pursue.

Another company, this one in the transportation industry, came to Palisade after finding itself in a disagreement with Oracle over how to count processor licenses in its virtual environment. "In this case we were talking about 60 processors. It was almost a $3 million judgment call," Guarente says. Ultimately the client prevailed. But, he adds, their legal counsel did so in a manner that did not destroy the business' relationship with Oracle.

Blake says Oracle is very transparent in terms of making all of its licensing practices, agreements and pricing available for public viewing online. But, like most hardware and software vendors, Oracle also exerts pressure on its sales force to increase licensing revenue growth for each account, which leads to "opportunistic behavior where they tend to interpret Oracle policies to their advantage," Blake adds.

Unfortunately, even though all of the documents are online, many customers remain unaware of their existence. "They fail to take the time to review them and take a stand," he says.

According to a VMware spokesperson, the overwhelming majority of vSphere customers choose to build separate clusters dedicated to Oracle-only workloads. But Welch says while some Oracle customers may decide to follow that path for business reasons, others may be doing so for what they see as contractual reasons. However, "if the customer decides that they want hundreds of vSphere servers in the fewest number of clusters possible, then it is their privilege to do so," he argues.

If you're not dedicating the cluster, and Oracle challenges you on pricing issues, Blake says, "My recommendation would be to . . . ask them to show you where in the contract you executed with them it asks them to pay a license fee for every server in the cluster."

The confusion over licensing isn't entirely of Oracle's making, says Guarente. In a virtual environment it's fast and easy to spin up a new Oracle server, and it's difficult for Oracle to confirm how many instances are running at any given time. The current environment needs to be compared to the bad old days of customers being forced to use activation codes and hardware-based dongles.

"Customers wanted flexible licensing and Oracle provided it. With that comes complexity," Guarente adds.

But that complexity is easily handled, says Welch. VMware's DRS Host Affinity Rules can restrict a virtual machine hosting Oracle to an Oracle-license vSphere sub-cluster, and there are other auditable ways to restrict a virtual machine to an Oracle-licensed sub-cluster.

Enterprises want to consolidate Oracle in virtual environments to reap the benefits of consolidation. On traditional server hardware, where utilization rates hover around 15%, IT typically over-sizes and over-licenses servers for peak loads. But with virtualization and related technologies such as Live Migration and vMotion, average daily server efficiency has climbed into the 50% to 60% range.

"You can license many fewer physical servers on native hardware," Welch says. A typical consolidation can reduce the number of physical servers that need to be licensed from say, nine servers to four, or even three. "That should be a massive savings," he says. But not only do some customers end up paying for new Oracle licenses on processors that aren't running Oracle in a vSphere cluster, they also end up paying up to a 22% annual maintenance fee even on licenses that have been retired.

Paying maintenance on dead licenses

Even if customers avoid paying for additional processor licenses after transitioning to a virtual environment, they may not be able to reduce ongoing software maintenance costs for licenses that are no longer in use after a server consolidation has taken place. That leaves many organizations paying anywhere from 15% to 22% of the original software license fee every year for products that just sit on the shelf.

So even if virtualization efforts reduce the number of licenses needed in a large organization by 30%, it can still add up to millions of dollars in potential annual support savings that go unrealized. "Oracle has set up virtualization in a way where users do not benefit from the consolidation," says Constellation Research's Wang. Some of his clients, he says, have moved to Microsoft SQL Server on VMware to save on licensing costs.

Oracle organizes licenses into license sets, says Welch. For example, all Oracle Database licensing tiers -- Standard One, Standard Edition and Enterprise Edition -- form a single license set. According to Oracle's matching service levels policy, you cant cancel maintenance and support on a subset of licenses within a license set without also cancelling those licenses. "Should the licensee desire later to use those licenses, they would have to purchase them from scratch, not just renew maintenance and support on them," Welch says.

Whats more, says Guarente, "If you terminate, then there's a re-pricing," and that doesn't always work in the customer's favor. For example, Oracle may remove previous discounts and raise prices, with the result that "the new price might be exactly what you paid [before]," he says.

Bill, the Fortune 500 company CIO, says no one should be surprised that they can't selectively cancel maintenance on individual processor licenses without canceling the entire contract and renegotiating. Package deals have offered substantial volume discounts, and when you go that route, "it's all or nothing," he says.

But contact prices have gone up, Bill acknowledges. His maintenance amounts have risen to 22% of the original license purchase price -- his contracts originally started at 15% -- and his discounts are much smaller than what was offered in the past, he says, though he declined to say by how much. So he's shifted away from big software contract packages in favor of making smaller buys and keeping those contracts separate.

"The more segregated your paper is, the better," Welch agrees. But many organizations have taken Oracle up on an offer to do a "migration" of their existing paper to pull everything together into one neat, tidy contract. "It's rarely in the customer's interest to do so," he claims.

Tom, the other CIO, has been down that road before. "It ended up being a zero-sum game," he says. "You're not going to get tremendous value out of it."

Fear factor

Many organizations don't want to make waves against the vendor of software on which their core business runs, especially one that has a reputation for being litigious. But Welch doesn't think customers should worry about pushing back, at least when it comes to vSphere clusters. "There's no case law anywhere where Oracle has gone after a customer for subcluster licensing," he says.

Guarente offers more measured advice. "I haven't heard of anybody being sued, but people are very afraid. Talk to your attorney," he says. Ultimately, he says, "My experience is that most customers buckle." His advice: "Don't go wobbly." There's simply too much money at stake.

For his part, Tom is cautious about pushing back too hard. "The next time you do any negotiation with them they're going to remember you pushed back on this, and it will come back to hurt you." Here's his strategy: "Try to figure out where you can extract concessions. Know when they're under pressure and when they're not. It's like a chess game. Individual moves don't win the game. It's your overall strategy that counts."

Customers want to be in compliance, and often the contract makes it clear when the customer should pay -- and for what. When it's unclear, however, there's room for negotiation, Guarente says. "None of this is in the contract. It's all in the policy documents out there. So you need to be an expert with Oracle and know when to push -- and when not to push."

This article, Virtualizing Oracle software: Don't pay for what you don't need, was originally published at Computerworld.com.

Robert L. Mitchell is a national correspondent for Computerworld. Follow him on Twitter at twitter.com/rmitch, or email him at rmitchell@computerworld.com.

Read more about virtualization in Computerworld's Virtualization Topic Center.