In thinking about the concept of agility in data warehousing, I got to wondering if it is possible to achieve agility and flexibility if it can’t be done quickly. I previously posted about building it right, but what about building it fast?
Think about something that is built in an inflexible or non-agile way, such as anything that is “handcrafted” – a piece of clothing, furniture, jewelry, musical instrument, etc. These handcrafted products typically sell at a higher price and take longer to produce. It may be due to more expensive materials used but most of it is the cost of labor and time put into creating the product. On the design side it often takes longer to respond – the “we’ve been making it this way for decades” phenomenon. For some products this is desirable and is a key selling attribute, but for others it is definitely not.
Data warehousing falls into the latter category. And yet we continue to lovingly handcraft data warehouses at great expense, time and resources. Changing our methodology is critical, but can it ever be fast enough to respond to changing business needs if it is still largely an exercise in handcrafting?
I read an interesting blog post by Chris Sorensen and I was with him until the very last line when he said “people, not software, enable agile BI.” In fact you need both. The right software plays a key enabling role to achieve a more agile data warehouse.
One way to accomplish this is to use a tool that automates tasks whenever possible. What kinds of tasks can be automated? How about: slowly changing dimensions, surrogate key management, suspense and exception handling, field mapping, incremental summary generation, task execution and monitoring, physical schema generation and management including staging tables for every object, sub-typing and inheritance, ragged hierarchies, generating and maintaining business metadata – to name a few. These are among the areas where IT spends many months or years on the initial implementation, and many weeks or months to deal with business change. Automation can save a lot of time and resources, and it also reduces errors by removing manual steps where they can too easily be made.
Here’s an example: one customer is a healthcare consortium that serves over 20 healthcare insurance providers and provides information and services to help their customers compete for national accounts. You can imagine the fast-paced nature of healthcare insurance these days! They are a small organization without a big IT staff. They needed a more agile data warehouse up and running quickly so they could analyze pricing, subscriber information and all the myriads of other data used to create a healthcare benefits program for a business, and provide that to their insurance provider members. A prior attempt using the traditional methodology and tools could not keep pace. But a switch to an agile methodology coupled with automated, agility-enabling tools has enabled them to not only have a more agile data warehouse, but be a more agile organization. You can hear this customer speak in a webcast about their project.
You need people AND software to enable agile BI.