Build BI tools fast with agile approach

Feeling like your business intelligence effort is a bit sluggish and out of touch with what the company needs? Maybe it’s time to try agile BI, a rapid development methodology that solicits end-user input early and often, and delivers BI systems fast.

While the use of the agile software development methodology is a big component of agile BI, it’s by no means its only attribute, says Boris Evelson, an analyst at Forrester Research.

Forrester recommends adopting multiple best practices and next-generation technologies to make BI more flexible. The research firm defines agile BI as an approach that uses processes, methodologies, organizational structure, tools and technologies to help strategic, tactical and operational decision-makers be more flexible and more responsive to the fast pace of change in business and regulatory requirements.

Before developing an agile BI strategy, a company must adapt its organizational structure and enterprise culture for agility, Evelson says, pointing out that no technology or process can address BI challenges if a company’s organizational structure and culture aren’t already designed to be agile. Once an organization is aligned for agility, it’s ready to adopt agile BI processes, he says.

Very few organizations have implemented agile BI as Forrester defines it, Evelson says. Based on anecdotal evidence and discussions with many clients, he estimates that out of all the organizations that use BI applications, probably less than 20% of the BI user population within those organizations is leveraging some kind of agile BI. But he predicts that that figure will climb to about 80% in the near future.

Despite the low adoption rate, companies in a variety of industries — including healthcare, retail, education, biotechnology and financial services — are already benefiting from agile BI, according to Evelson.

A Good Fit

Business intelligence is an enterprise application that is particularly well suited for agility and the agile development methodology, says David White, an analyst specializing in BI at research firm Aberdeen Group.

Research that Aberdeen conducted in 2010 shows that despite the wealth of experience that many end-user organizations have in implementing BI systems, only 43% of business intelligence projects were delivered on time or early. Many companies still struggle to deliver the right information to the right business managers at the right time.

Research conducted by Aberdeen in February and March this year indicates that organizations face three significant challenges when it comes to effectively delivering BI that is truly valuable to the business.

One is that data volumes and the number of BI data sources are growing. Another is that the amount of time managers have to make decisions is shrinking. And a third is that demand for management information is always either increasing or changing.

Clearly, BI implementations have to be more agile so managers can easily find the information they need as business requirements change.

One of the most important steps companies must take to ensure that their BI implementations are agile is to analyze their organizations’ business needs and take those into account when BI software is being configured and deployed, White says.

The research Aberdeen conducted this year shows that organizations with the most agile BI implementations are more likely to have processes in place for ensuring that business needs are being met.

To determine which organizations have the most agile BI systems, Aberdeen evaluated factors such as the availability of timely management information, average time required to add a column to an existing report, and average time required to create a new dashboard.

Another key to success is getting end users heavily involved in the BI development process. At nearly 70% of the organizations that are successfully deploying agile BI, IT and business people collaborate frequently, according to Aberdeen. Such interaction is less common at organizations that are less agile; only 50% of them report frequent collaboration between IT and the business.

The close collaboration that’s needed might not come easily, White notes. “There has always has been a communications barrier between IT and the business,” he says, and that barrier must be broken if agile BI is to become a reality.

White cites the development of a new dashboard to illustrate how collaboration might work: An IT developer planning a prototype dashboard could ask business managers what they need in the dashboard, develop a version, then go back to managers to get feedback — in some cases multiple times.

“That close involvement and the iterative process is part of the agile methodology, and it’s very applicable to BI,” White says. “Developers working with business users side by side or close by is very powerful.”

Rapid Delivery

Organizations using agile BI are tapping into that power. One example is Kiva, a San Francisco-based nonprofit that arranges person-to-person loans via the Internet to help alleviate poverty. In 2010, Kiva implemented a new data warehousing and BI application based on Pentaho’s Agile BI technology to handle its rapidly growing stockpile of data and replace an increasingly inefficient ad hoc analysis process that was done via spreadsheets.

Kiva has grown rapidly since its founding in 2005, and it has helped entrepreneurs around the world secure more than $200 million in microloans from individual lenders. As of April 2011, the organization had brought together more than 570,000 lenders, 131 microfinance institutions and more than 450 volunteers around the globe.

The nonprofit has used Pentaho’s open-source development platform to create monthly, weekly and program-specific dashboards that product managers and finance specialists can use to track and manage key metrics such as transaction volumes, new user registrations and promotion performance, says Greg Allen, business analyst at Kiva.

The organization has developed tools such as a “loan posting explorer,” an application that allows risk managers to analyze loan posting patterns based on region and country, type of loan and type of borrower, he says.

The biggest benefit that Kiva’s agile BI efforts have yielded is the ability to rapidly and incrementally deliver information to end users. “We have a big vision for our data warehouse project, which will require a substantial time investment to complete,” Allen says. “Instead of forcing the organization to wait for a ‘big switch,’ we are able to focus on specific business areas to design, create and release data marts and reports as part of our iterative development.”

Documenting user requirements is critical to the design of the data structures underlying Kiva’s BI system, Allen says. “We’ve found that our users respond best to testing early versions of our BI solutions,” he says. “Ad hoc data sources enable us to test business logic to collect and incorporate user feedback quickly, before developing more robust reporting data structures. This can expose unforeseen [difficulties and] lead to more comprehensive solutions.”

Allen says having greater agility with BI development gives the organization more flexibility. “Our business can change quickly, and being able to respond quickly with data-driven decisions is critical to our growth,” he says. “Agile BI enables us to capture new data and transform it into information.”

Kiva handles millions of financial transactions, and accuracy and consistency are key. “Being agile means finding ways to break large, complex solutions to reporting challenges into iteration-sized chunks so we can measure progress along the way,” Allen says.

Having end users involved in the process is vital. “Our users push us to include leading-edge data sources,” Allen says. “While being agile allows us to respond to those needs quickly, it also sets up the potential for significant reloading time if refactoring is needed. For example, what started as a single large table is evolving into a more normalized schema. [Collaboration] and setting expectations with end users is critical in these cases.”

But there can be challenges with agile BI development. “Large BI projects can get stalled living in internal cycles focused on minute design details,” Allen says. While business users are given regular progress updates on these projects, he says, they might not see benefits right away.

Although Kiva is at the beginning of an ambitious plan for its data warehouse and reporting capabilities, “we are already able to deliver new insights on our user base and partners,” Allen says. “Often, these early or narrow views into business areas have generated new questions that improve the next iteration of development.”

Violino is a freelance writer in Massapequa Park, N.Y. You can contact him at bviolino@optonline.net.