Wednesday, October 29, 2014

Quality of Big Data – Turning Volumes into Insights

Executives love to talk about big data and the amount of data being collected. However, big data is more than just the big talk of the town. Wall Street is now planning to assign a dollar value to the data assets of enterprises. Everyone presumes that big data can make businesses more agile and responsive. But the reality is that within a few months of launching big data projects, managements realize that they need more fundamental changes in organizing and analyzing this big data, irrespective of the technology platforms. Unless a strategy is applied to find deep insights from this data, and in a more natural way, it is unlikely that the efforts invested to harvest big data will reap long term benefits.


After a series of meetings with multiple customers recently, I realized that the different business units in the organization do not have an agreement on a common definition of customer. While it seems basic to have a common lexicon, the reason for discrepancy is that the value of the data depends on the use case and what it is used for. Though most people like to talk about good data, the bad data of today can become good data for tomorrow, and there is no perpetuity for the value of today’s good data. 

While the social and mobile platforms have been generating volumes of data, it is important to note that the variety of data sources, aka islands, are also vastly increasing. Aggregating these different sets of information is the key to arrive at better insights. For instance, imagine that the sales data is in one data source, and that the customer loyalty information is in another. How do you easily find the information about the stores that get the most number of repeat customers?

It is important to note that as long as the data is not made available to those who make decisions, it will be deemed as unusable, aka data noise. Traditionally, data resides only with IT, and businesses never get to play around with the data to find the deep insights. Common sense will make you laugh at this problem – but that is how all the enterprises have been. Businesses and IT just point fingers at each other on where they are and think that it is not their problem to solve. Most IT administrators manage either the infrastructure or the tools required for data rather than managing the data itself.

The source of truth ends up as a questionable topic in most organizations since every business transformation project or M&A activity introduces changes to data infrastructure, operations, and underlying business applications. Depending on the organizational maturity, either the changes are not consistently applied or different businesses remain with copies of out of sync data.

In order to realize the true insights from the big data and make it a good data, it is important to identify a business function or a set of problems and opportunities to get this off the ground and stay focused. Instead of focusing too much on the data governance or data definitions, get technologies that can drive up the data usage by the business teams. The usage would in turn boost the feedback and inputs for governance and data management. The key is to have an ongoing Data Quality program, driven by data usage that monitors constant improvements. Given the rapid growth rate of data, there will never be a single system of truth. Instead, consider having a single system of reference which becomes the single reference point for analyzing data and getting business insights.

There is no question that data is one of the most valuable strategic resources for any organization, and big data presents a great opportunity to leverage it. The strategy should not be focused on implementing big data and analytics platforms. But the real strategy is to identify the new white-space opportunities and customer insights that big data (along with analytics) will make available.

(Picture credit: CIO Insight)