This vendor-written tech primer has been edited by Network World to eliminate product promotion, but readers should note it will likely favor the submitter's approach.
Big data is getting a lot of coverage of late and with good reason. We live in a world that is fast becoming overwhelmed by information. Ninety percent of the world's data was created in just the last two years. From software applications and social media to Internet search results and the ever-present email, the rate of data creation is growing exponentially with no signs of slowing.
This has resulted in somewhat of a "dog catching the car" scenario. Companies have for years seen information as the holy grail of competitive differentiation -- if only we had more customer data, if only we knew more about market patterns, if only our equipment could tell us when it was going to fail. In the blink of a cosmic eye, we have gone from thirsting for this information to drowning in it, leaving us to ask the question "now what?" Many see big data as the answer -- and the key to making "if only" a reality.
[ ANALYSIS:Big-data science requires SDN, Internet2 chief says]
Despite all the focus, the actual meaning of big data and its application aren't always clearly articulated, but the concept is actually a fairly simple one.
At the most basic level, big data is just as it sounds -- volumes of information that are extremely large and growing. That can mean data already being generated within an organization by systems or machines, like manufacturing output or customer buying patterns; or increasingly, data that's available for companies to buy, provided from sources like social media vehicles or Internet search providers.
The problem comes when the volume gets so big that it can't be effectively managed in a traditional "batch processing" way. If it takes many hours or even days to process large volumes of data, the information quickly loses its value. Enter big data solutions designed to turn this information tsunami into an information gold mine.
Because of this potential and all the hype, companies understandably want to make sure they don't miss out on the big data phenomenon. Truth is, we're far from reaching a point where more than a small percentage of businesses truly have a big data problem.
Big data opportunities, on the other hand, are a zebra of a different stripe. Certainly, the possibility of making information a competitive advantage is a real one, more viable in fact than it has ever been. The question is whether big data holds the key to achieving that goal. For some, the answer will be a resounding yes. For others, however, big data could end up being a costly and unnecessary distraction.
How do you know where your company falls? Some of the best advice about how to address the big data question can probably be summed up by admonitions that we frequently give to our children:
* Set the table. The first step in any data project (big or otherwise) is to clearly establish what you're trying to accomplish. Additionally, it can be helpful to establish a visual picture or framework that organizes the business around an end game and breaks down the problem into consumable, achievable initiatives. With all the excitement around the big data concept, it's easy to get pulled into an initiative with objectives that are likely to change as you go along. Defining upfront what you need helps determine if big data is the answer and puts parameters around what you undertake if it is.
* Eat what we already have in the pantry. Many companies have valuable data ready and waiting to be exploited with the help of business intelligence and analytical tools. Before assuming that an expensive initiative around big data is required, determine what data you have and whether you can accomplish your goals with existing tools. Incremental capabilities can also be incorporated in a non-disruptive way, giving you a big bang for your buck by providing better, more consistent visibility to data and making it more consumable. [also see: "10 Mobile Business Intelligence Apps for On-the-Go Analysis"]
Analytic applications that surround existing information and transactional systems can effectively manage your data while also delivering a host of other benefits, like giving non-technical users simple dashboards that help them target specific problems and opportunities; delivering information directly and automatically to users so they head off problems and make better decisions; and making collaboration a standard part of your work processes.
* Finish what's on your plate before asking for more. All of the new data available today can be tempting. From Amazon to Facebook, we now have a dizzying array of choices that seem poised to unlock new and previously untapped opportunities. However, the data you already have can be equally as valuable, if not more so, because of its relevance and quality. At some point, there are diminishing returns on adding more insight into the mix, as too many data sources can create "analysis paralysis," particularly for the average worker.
Improving your ability to deliver a highly focused view of the business with targeted workflows is likely to serve you better than focusing time and energy on what can easily turn into a great deal of outside noise. The best approach is to make sure you fully understand and exploit what's core to your business and already exists before taking on new, potentially complex and expensive data initiatives.
* Take small bites. When you are ready to start delving into big data, start with a prototype and refine your processes before undertaking a major initiative. Once you have standard approaches in place that capitalize on your internal data and support consistent, streamlined business processes, take a small piece of the business and inject one or two new data sources to see if the benefits outweigh the cost and distraction. Testing the waters -- and the results -- will help you keep things under control and better assess the true potential return on big data for your business.
Certainly, big data has the potential to deliver big value. But the size of that potential and the most effective ways to exploit it are still being explored. Companies that watch, plan, and act when the time is right will be well served.
Read more about software in Network World's Software section.
