Maybe it’s time to apply Agile principles to Data Warehouse
My colleagues and I have recently been searching for a the perfect Reference Data Model for Financial Markets. The reason for the search is to establish the common data model for our enterprise so a Data Warehouse can be built that will not have to change.
Like every other bank, we already have a Data Warehouse – or two… But like most existing Data Warehouses, it does not meet all requirements and it is too difficult to change. Thus the natural inclination is build another.
Thinking a bit differently, the existing Data Warehouse at one point probably did meet business requirements. But, perhaps the real issue is not that the model was wrong from the start. Rather, the need for rapid painless change was not engineered into the solution.
Much of the motivation behind Agile software development methods is to accommodate change. Change is a constant. If we accept there will always be change and we adopt methods that mitigate the risk and cost of change in our systems, we can reduce the time we spend on Analysis and Design. Today we spend inordinate amounts of time on Analysis and Design because we have all been taught that Change is Costly so we better get it right up front. This places tremendous stress on Business Analysts and Architects to produce extensive documents.
In the Data Warehouse domain, change to the data model of the Data Warehouse looks extremely painful because it affects so many things. So, naturally we aim to get the data model right up front. In the current project I’m looking at, the immediate requirements are not so complex. However, because we want get the data model right for all future requirements we have made the analysis task massive.
But yesterday I woke up and it (literally) dawned on me that we need to apply Agile principles to this Data Warehouse project. But, I’ve only applied Agile to Software Development. I really was not sure what this meant.
Agile Data Warehouse
If you’ve got a little time it doesn’t take long to find what you’re looking for. I’d come across Scott Ambler’s writing on Agile before and good old google reminded me by bringing the Agile Modeling and Agile Best Practices for Data Warehousing essay is just what I was looking for.