In my reading I recently came across the concept of the Data, Information, Knowledge, Wisdom (DIKW) hierarchy. While it is perhaps loosely defined, I see the concept as a helpful way to illustrate the progression from raw data to actionable information (requiring wisdom to handle it correctly). There are valid criticisms of this model, but it does reflect the goal of collecting and managing data - turning it into something that allows business users to make wise decisions.

So how do we get our business customers from data (which is available in quantities so big they are difficult to grasp) to knowledge and wisdom? Let's consider the illustration of buying a house.

Home buyers in 2011 have more data at their fingertips than ever before. Home buying websites like Zillow provide prices of recently sold homes, homes for sale, and estimates of homes that are not for sale. Mortgage websites like Bankrate, Lending Tree, and many others supply daily quotes of mortgage rates of all varieties. There is even a large supply of information about buying a house: lists of do's and don'ts, ratings on realtors, personal experiences filled with suggestions and caveats.

With this glut of data and information, surely home buying must be far easier than it was before. Perhaps, but there are some new challenges to deal with as well. With all the readily accessible data and means for capturing it, there often comes an implicit assumption that more is better and will inevitably lead to increases in knowledge and wisdom. My experience with both home buying and data warehousing is that it is easy to overstate the connections between data and knowledge (or "business intelligence" in industry terms). In both cases, hard work is required to harness raw data, creating solid information that leads to better decisions.

For example, the discerning home buyer quickly realizes that while Zillow's home estimates are a helpful starting point, it takes many visits to prospective homes to get a real feel for valuations and for what different price points can buy in the market. Similarly, we would do our business customers a disservice if we turn over a shiny new data warehouse with no training, explanations, and interactions to show what is available and to tweak things to give them what they need. In fact, needs often become clearer as business customers start to use the reports and querying capabilities of their new data warehouse.

Additionally, we need to highlight the importance of quality in ascending the DIKW hierarchy. With the availability of so many websites that provide real-time mortgage rate quotes, it should be easier than ever to shop around for the best rate. But in fact, the quality of this rate information is often suspect, and sometimes insidious. Some lenders use rate sites to advertise impossibly low teaser interest rates that they have no intention of actually securing for a borrower. Instead, they hope to lure in unsuspecting borrowers at the teaser rate, and then to gamble on either getting them the rate via a future rate drop or dragging the home buyer so far down the loan application process that there is no turning back when the rate accidentally is higher than their initial promise.

Data quality issues internal to businesses are rarely this malevolent, but the damages can be just as costly. We need to work with business customers to be sure they have correct data organized in a way to provide them with the correct information. We should highlight data quality issues that prevent this from happening. While 100% accuracy may be cost-prohibitive, cleansing key data can go a long way to transforming it into actionable information.

Though data warehousing and its related disciplines cannot guarantee business customers can go from having raw data to having knowledge and making wise choices, we must help address the challenges as we work together to move up the DIKW hierarchy. And watch out for unscrupulous lenders!