The first twenty-plus years of data warehouse projects yielded failure rates above fifty percent, and the waterfall methodology took much of the blame. Ironically, as Ralph Hughes points out in his book Agile Data Warehousing Project Management, the waterfall methodology itself was based on a “grievous misreading” of another work. Dr. Royce’s 1970 white paper Managing the Development of Large Software Systems had called for a more iterative approach to systems development than the circa-1985 waterfall methodology it inspired.
The need for a more iterative approach to data warehouse projects was clear, and agile implementation methodologies had become mainstream by the mid-2000s. But the question remains: Is your organization delivering faster using an agile approach? Ralph Hughes and other experts estimate that, “A new team should be able to achieve a 20 to 40% improvement in development speed within the first year of operation.” If your agile development efforts haven’t delivered those results, you’re not alone.
There are many reasons why organizations aren’t getting great results from their agile warehouse projects, but it mostly boils down to organizations developing the same way they always have — only with smaller teams! Here are some suggestions for getting those agile warehouse projects out of second gear.
- Stop trying to think of everything up front. Ralph Hughes advocates “80-20 specifications”, which means good enough to retain clarity while remaining flexible. Over-engineering the design won’t change-proof it, but it will create delays and rework.
- Stop documenting during development. Heavy investment in ‘as-designed’ documentation slows down development — or requires teams to work longer hours. A better approach is to focus on ‘as-built’ documentation after a sprint has been delivered and accepted.
- Create agile role models in your organization. Start with one or two teams staffed with individuals who embrace new ideas and don’t mind if requirements seem semi-baked when they start developing.
- Equip the team with agile tools that help them keep pace with development sprints. Tools that automate tasks and make quick work of rework are essential to delivering the most important metric for agile developers — a normal working cadence.
Agile can lead to team burnout if the sprint concept is taken too literally. Burnout and disappointing results are inevitable if teams use the same processes and tools as before — except now with less time to get it all done.