MDM and data quality for the data warehouse

A recent Information Management article raised a number of issues with data warehouses and why they are such time-consuming projects. According to the article, the main reasons are primarily around changing scope, data quality and ETL design. I’ve discussed how to handle the scope and design issues in earlier posts about business modeling. Over the next few posts, I’ll talk about how to deal with data quality in the data warehouse. Read more

Who in the world needs a data warehouse with bad data in it?

I just read a recent article in Information Management entitled “Who in the World Needs a Data Warehouse?” On seeing the title, I immediately thought this would be an article about how companies can simply load their data in memory to avoid building a data warehouse entirely. As I suspected, that is the author’s proposed option. The article listed off a variety of issues with building a data warehouse, including capturing requirements and dealing with scope creep, ETL design complexity, data integration and data quality. In several prior posts, I discussed ways to handle capturing requirements and the ETL through business modeling and automation. But in this post I want to discuss data quality. The article says “data that is housed in the data warehouse is often either incorrect or inconsistent.” If in-memory analytics is the answer to this problem, I ask: what’s the difference if you load incorrect and inconsistent data into memory? How does simply moving it off disk fix this problem? The answer is: it doesn’t! Read more

Managing Master Data Using Federalist Principles

In the early days of MDM, a common question was, “union or intersection?” Business Unit A has 30 attributes for customer. Business Unit B has 50. 10 of them are common. The question was, in other words, “should MDM manage all 70 attributes, or just the 10 common attributes?” Read more

Data Stewardship for Data Warehousing and Analytics

The purists’ view is: a data warehouse integrates and stores data sourced from operational systems. In other words, a data warehouse does not create new data: it only combines and repackages data created elsewhere for analysis. Read more

How to Keep the Enterprise Data Warehouse Relevant

Last week, a data architect from a large pharma asked me: “Should we build an enterprise data warehouse given that we want to harmonize business processes globally?” My first reaction was: it’s been a while since I heard anyone wanting to build a true EDW. Read more

Data Governance is a Prerequisite for Effective MDM and Data Quality

Dave Waddington is senior vice president and co-founder of The Information Difference, an analyst firm specializing in data quality, master data management (MDM) and data governance. He is a recognized authority and practitioner in the field of data management and has advised both vendors and corporations on data management strategy and architecture. Read more

How to Measure Data Accuracy?

(Updated October 2018) If you believe that better data quality has huge business value, and you believe the old axiom that you cannot improve something if you cannot measure it, then it follows that measuring data quality is very, very important. And it’s not a one-time exercise. Data quality should be measured continuously to establish a baseline and trend; otherwise continuous improvement wouldn’t be possible. Read more

Forrester’s thoughts on data governance and business process

This morning, I read a great blog post by Rob Karel of Forrester. He talked about the need to increase focus on business process to build momentum for data governance. I couldn’t agree more. Most data governance programs consider data AND business process AND organization; it’s the only way to provide the proper level of context to defining how data is used across the enterprise. Read more

Can we make it real?

The single biggest challenge for any software company is deciding what should go into their products. As we thought about the data governance, we considered the options; do we build on top of our existing MDM product or start from scratch? Most of the vendors we compete with in the MDM market were planning on extending MDM, but we felt this was a flawed strategy. Why? Because MDM is repository-centric and focused on managing master data only. We believed that in order to build an effective data governance product, we needed to consider ALL data, as well as business process and organizational context. Read more

It all started in August 2008

August 20, 2008. It started on this day. As I thought about the future of Kalido, I wrote down what I thought were it’s core assets; it’s people, it’s customers and our ability to support better data throughout the enterprise. I’d recently read some great things from Gwen Thomas at the Data Governance Institute, and as I learned more about data governance, I realized what a great fit it was for Kalido. We’ve always managed and tracked data at multiple levels; we understand the workflow necessary to support changes to data, and our business model makes us very business friendly. Read more