As I was straightening up my home office this weekend, I removed a stack of notes from on top of a stack of “go to” books on my desk. Three of those books belonged to “The Data Model Resource Book” series from Len Silverston (a great source of reference for data warehouse modeling patterns across various industries). Just seeing the books reminded me of a conversation he and I had a few months back while comparing scars from the data wars we’ve each been a part of over our careers.
During the conversation Len brought up the challenges brought on by the complexities of data mining. But, he quickly explained that he didn’t mean mining as in looking for insights hidden in the patterns of data. He meant; “that data is mine and you can’t have it!” The mistaken notion of data ownership by IT or pockets of business that fail to recognized that the whole is greater than the sum of its parts or that hoarding some piece of data or business knowledge makes you more valuable to the organization. Misconceptions such as these pose the biggest challenges that we as data management professionals must overcome – and they are cultural not technical
How much time is spent during a project to find some necessary piece of information or in convincing someone to share their data? Pockets of tribal knowledge and private stashes of data continue to plague organizations large and small – despite the known risks of these ungoverned repositories or undocumented procedures.
On a recent project we had a room full of no less than 15 people to answer a very simple question about a balancing process for a core transactional system. Each question yielded a very specific piece of data that led to the next question which had to be answered by someone else – often times the person wasn’t in the room and had to be called in to get us to the next step. The “mine-ing” in this case wasn’t so much the data as it was the “proprietary” nature of each person’s part in the process. Nobody had a good overview of the entire process making it painful (and lengthy) to extract the full set of logic required to reconcile our data loads against the definitive source. And, in an agile data warehouse implementation, lengthy can be costly.
What we now refer to as the “marathon” meeting was a valuable lesson to all. For the client it highlighted a missing piece to their eco-system – knowledge sharing. For my project team it clearly defined the need to provide adequate notice to subject matter experts about the business issue to be resolved in the meeting so everyone can come prepared and not to overestimate the breadth of knowledge than an SME may have. While all participants were subject matter experts, none were experts in the overall subject.
We’re already assembling the set of questions to manage reconciliation for our next subject area and constructing an overarching process description so that we can pose it to the SMEs well beforehand (we’re not scheduled to get to the new source for several weeks yet). My goal is to reduce the number of people and the meeting time required to answer the question by at least half. I’ll let you know how we did.