Off to a Fast Start

Feeling like your business intelligence efforts are a bit sluggish and out of touch with what the company needs? Maybe it’s time to try agile BI, a rapid development methodology that solicits end-user input early and often and delivers BI systems fast.

While the use of the agile software development methodology is a big component of agile BI, it’s by no means the only attribute, says Boris Evelson, an analyst at Forrester Research.

Forrester defines agile BI as an approach that combines processes, methodologies, tools and technologies, while incorporating organizational structure, in order to help strategic, tactical and operational decision-makers be more flexible and more responsive to ever-changing business and regulatory requirements.

Very few organizations have implemented agile BI as Forrester defines it, Evelson says. Based on anecdotal evidence and discussions with clients, he estimates that out of all the organizations that use BI applications, probably less than 20% of the BI user population within those organizations is leveraging some kind of agile BI. But he predicts that this figure will climb to about 80% in the near future.

Business intelligence is particularly well suited for agility and the agile development methodology, says David White, an analyst specializing in BI at research firm Aberdeen Group.

Research conducted by Aberdeen in February and March indicates that organizations face three significant challenges when it comes to effectively delivering BI that is truly valuable to the business.

First, data volumes and the number of BI data sources are growing. Second, the amount of time managers can devote to decision-making is shrinking. And third, demand for management information is always increasing.

Clearly, BI implementations have to be more agile so managers can easily find the information they need as business requirements change. And to achieve that, companies must analyze their organizations’ business needs and take them into account when configuring and deploying BI software, White says.

The Aberdeen research shows that organizations with the most highly agile BI implementations are more likely to have processes in place for ensuring that business needs are being met.

Another key to success is to heavily involve end users in the BI development process. At nearly 70% of the organizations that are successfully deploying agile BI, IT and business people collaborate frequently, according to Aberdeen. Such interaction is not as common at organizations that are less agile; only 50% of them report frequent collaboration between IT and the business.

The interaction that’s needed might not come easily, White notes. “There has always has been a communications barrier between IT and the business,” he says, and that barrier must be knocked down if agile BI is to become a reality. “That close involvement and the iterative process is part of the agile methodology, and it’s very applicable to BI. Developers working with business users side by side or close by is very powerful.”

Rapid delivery

Organizations using agile BI are tapping into that power. One example is Kiva, a San Francisco-based nonprofit that arranges person-to-person loans via the Internet to help alleviate poverty. In 2010, Kiva implemented a new data warehousing and BI application based on Pentaho’s Agile BI technology to handle its rapidly growing stockpile of data and replace an increasingly inefficient ad hoc analysis process that was done via spreadsheets.

Kiva has grown rapidly since its founding in 2005, and it has helped entrepreneurs around the world secure more than $200 million in microloans from individual lenders. As of April 2011, the organization had united more than 570,000 lenders, 131 microfinance institutions and some 450 volunteers around the globe.

The nonprofit has used Pentaho’s open-source development platform to create monthly, weekly and program-specific dashboards that product managers and finance specialists can use to track and manage key metrics such as transaction volumes, new user registrations and promotion performance, says Greg Allen, a business analyst at Kiva. The organization has developed tools such as a “loan posting explorer,” an application that allows risk managers to analyze loan posting patterns based on region and country, type of loan and type of borrower, he says.

The biggest benefit that Kiva’s agile BI efforts have yielded is the ability to rapidly and incrementally deliver information to end users. “We have a big vision for our data warehouse project, which will require a substantial time investment to complete,” Allen says. “Instead of forcing the organization to wait for a ‘big switch,’ we are able to focus on specific business areas to design, create and release data marts and reports as part of our iterative development.”

Documenting user requirements is critical to the design of the data structures underlying Kiva’s BI system, Allen says. “We’ve found that our users respond best to testing early versions of our BI solutions,” he says. “Ad hoc data sources enable us to test business logic to collect and incorporate user feedback quickly, before developing more robust reporting data structures. This can expose unforeseen [difficulties and] lead to more comprehensive solutions.”

Allen says having greater agility with BI development gives the organization more flexibility. “Our business can change quickly, and being able to respond quickly with data-driven decisions is critical to our growth,” he says. “Agile BI enables us to capture new data and transform it into information.”

Although Kiva is at the beginning of an ambitious plan for its data warehouse and reporting capabilities, “we are already able to deliver new insights on our user base and partners,” Allen says. “Often, these early or narrow views into business areas have generated new questions that improve the next iteration of development.”

Violino is a freelance writer in Massapequa Park, N.Y. You can contact him at bviolino@optonline.net.

This version of this story was originally published in Computerworld‘s print edition. It was adapted from an article that appeared earlier on Computerworld.com.