I don’t know if you have ever used the Philips head screwdriver on your Swiss Army Knife, but if you have you know that it is only with muscle, time, dexterity and bruised knuckles that it gets the job done (most of the time). And if you’ve had this experience you also know how much quicker, easier and less painful it would have been to use a screwdriver, or better yet, a power drill, especially if you are trying to join two pieces of hard wood. Now, the Swiss Army Knife is a fabulous tool, but it is limited by the need to place so many disparate implements in such a restricted form factor.
It is the same way with a data warehouse (DWH). DWHs are quite valuable from an enterprise and line-of-business (LOB) viewpoint to enable understanding of what-has-been as tracked against specified goals and/or what-has-changed as a result of corporate initiatives. By definition, DWHs are subject-oriented, integrated, non-volatile and time-variant. While gurus quibble over the precise meanings of these terms, the DWHing community generally recognizes them in the following contexts.
- Subject-oriented refers to the fact that a DWH must warehouse data of a particular kind, whether sales data, enterprise data, shipping data, manufacturing data, etc.(;)
- Integrated specifies that although the data is of a particular kind, it is pulled from many different source systems and brought together in the DWH according to matching, alignment and enhancement rules;
- Non-volatile is taken to mean that data once integrated and entered into a DWH is not to be changed in either value or in relation to other data;
- Time-variant is a fancy way of saying that DWHs contain historical data so you can track changes in subject-oriented, integrated and non-volatile data over time.
Sounds great! Who wouldn’t want a DWH? But wait, there are inevitable consequences in the combination of these attributes of a good DWH. As a DWH changes in depth, breadth and/or structure of content, the benefits of subject-oriented, integrated, non-volatile, and time-variant data become limitations. And as the speed and magnitude of change in the DWH increases, these limitations rapidly become strictures.
In particular, it is nearly impossible to maintain an integrated, non-volatile DWH in dynamic business environments where business processes and architectures are evolving to seek out opportunities in changing, highly-competitive markets and/or in dynamic data environments where data sources are in flux with additions, deletions, corrections, upgrades and rearrangements. The rate of change simply exceeds the capacity to identify, integrate, test and curate new data and new data rules while rectifying them with the existing DWH data, structures and rules. The effort that must go into this change control and impact analysis requires time and skill, just as turning a screw with a Swiss Army Knife requires muscle and dexterity.
Furthermore, the subject-orientation of a DWH has baked into it a pre-defined model of the subject and a set of expectations about that subject. What-If and What-Might-Be analyses require flexible models and extremely rapid iterations as these are exploratory analyses and thus require much tinkering and experimentation. Pre-defined models and their attendant expectations are the form factors, which limit the functionality and flexibility of a DWH.
Finally, with each new data set to be integrated and each new reporting requirement to be accommodated, the overall utility of the DWH actually begins to decrease. Just as there is a practical limit to the number of disparate tools that can be functionally useful in a Swiss Army Knife, there is a limit to the number of data sets and reporting requirements that can be functionally integrated into a practical DWH model.
Yes, DWHs can provide real value. DWHs are good at identifying and tracking long-term market trends as well as the impact of corporate initiatives; at generating consolidated reporting at the Enterprise or LOB level; at providing the raw material for data mining; and at finding missed or hidden opportunities in existing business processes which are ripe for immediate or near-term capitalization.
If, however, you need to understand what is happening in your business right now or how a change in your organization might affect your business tomorrow, DWHs offer limited capabilities. Your business might be dynamic or your data in flux. These scenarios require agile reporting and analytics. Applying a DWH would be like the Philips head screwdriver on your Swiss Army Knife – it will be slow, difficult and painfully susceptible to slippages and failure.
-Marc d. Paradis, SM
Originally Published June 30, 2011 on Trinity Pharma Solutions (subsequently Shyft Analytics before being purchased by Medidata in 2018 and then Dassault Systèmes in 2019) Blogsite.
This article can also be found on Marc d. Paradis’ and SIYOM Consulting’s LinkedIn posts
Disclaimer: The opinions expressed herein are my own personal opinions and do not represent any of my previous employers’ views in any way.