We've recently collected a selection of our blog posts and videos related to reference data into a spotlight on our web site. A number of these resources are now six or seven years old, but upon revisiting them we thought they were still valid and with any luck useful to readers or viewers. But this exercise inspired us to think about this topic again.
Without restating too much that's already available, reference data can be understood as reusable data that modifies, describes, or enriches other data. The most common examples tend to be codes (often, but not always, some form of abbreviation) that tell us something about an entity. What's this person's employment status? What is the procedure type for the treatment this patient is receiving? What kind of financial asset is this thing? Most data applications store these reference data sets as lists of code values and descriptions, and when updating some other kind of record a user can choose from the values in an appropriate list. It doesn't have to be this way, but most of the time the code - a short value - is copied to the entity's record, while the description for that code lives in the lookup or validation table.
Reference data supports consistency, it helps to avoid error, and probably historically it was a way to save time and effort by using fewer keystrokes and saving fewer bytes of data. But it's easy to see where breakdowns could occur. For example, someone's given me a list of patients in my clinic right now and the procedures they're here for, but they gave me the codes without the corresponding description. What does XP242 mean? Broken femur? Nosebleed? Cancer? What happens if my list of codes is easier to understand, but someone adds a new one? I know what A, B, C, and D stand for, but what's this E all about? A surprisingly common situation reported by our clients, even today, is data products that map and/or aggregate records across multipe codes, and which basically stop working when changes to underlying reference data are made. Codes A and B map to 1, and codes C and D map to 2; everything else gets ignored because until recently we only had A through D!
While we haven't seen it too much, we have heard about issues confusing master data and reference data. Master data, to oversimplify, has to do with those entities and transactions that are updated or modified by reference data. Some information about those employees or those patients is master data, and while that's surely going to start with biographical or demographic information, the full scope of master data will vary by organization. Many of the challenges with master data have to do with its passage across systems and/or between departments. Some departments will have different standards for completeness, or around verification. Data applications will vary with respect to what information can be captured, how it's stored, how easy it is to include in integration processes.
It's not too hard to believe that organizations haven't developed good processes to communicate and manage updates to reference data or changes to master data, much less take proactive steps to discuss why these changes are needed and what impact they have. Who's going to recognize this as a organizational governance problem, rather than just another inconvenience caused by some other unit? And, even if it is recognized as a governance problem, what can be done about it?
From a distance, these seem like pretty simple problems to solve. In system X we capture a certain amount and kind of information about a person, organization, transaction, what have you. We need to send that data to system Y, but maybe not all of it. And we need to massage the data as it comes out of system X in order to conform with the way system Y is configured. What's so hard about that? Just write down all the rules, do all the testing to make sure your processes follow the rules, and move on!
But, as anyone who's spent any time at all in the space can attest, things are never that simple. We've talked about this in other posts and our webinars, but it bears repeating. Data stewards have their hands full managing the volume of data transactions in front of them, plus they're expected to answer questions about the data they control, and to keep an eye on data quality and on requests for access and on the laws and regulations that apply to their area. It's asking a lot for them to be informed about what happens to data when it moves downstream, and it's asking even more for them to change their business process simply to avoid someone else's potential inconvenience!
The people designated as data stewards are probably functional leaders, although some stewards are operating stewards, or technical, or people whose stewardship activities focus on compliance or accountability. Even enterprise stewards, who are expected to keep an eye on data practices across the organization, are unlikely to get down into these weeds, and where they do their focus will almost certainly be some form of master data management.
It's certainly good to have a policy that governs how and when reference data is changed, and who needs to be involved in those decisions. People will make good-faith efforts to abide by such a policy, but there will come a time when someone can't attend a meeting, or wasn't attentive during the meeting, or too many emails came in that day, or some external agency forced us to circumvent our normal practice. So it's also good to have some kind of process in place to check for changes like these, and for that process to include steps where interested parties get notified. We should also recognize that sometimes reference data gets updated without any significant downstream effect. The situation is not always dire!
About that downstream effect. Data products get stale: reports include time periods we no longer care about, dashboards present out-of-date KPIs, integrations don't include new data points or don't contain logic reflecting new endpoints, ETL error-checking doesn't get reviewed, and so on. Governance is needed throughout the data life cycle, and perhaps even more strongly at the point in the cycle where less savvy consumers are most heavily involved. Certainly this point represents a prime opportunity for the kind of engagement with data that can be most fruitful, whether that's variations in usage and terminology, assumptions about data provenance and quality, differences of opinion about analytical methods, or any other question that might arise.
As with most other data governance and data intelligence challenges, in our experience the solution does not involve placing blame, writing new policy, or even creating more technical oversight. Now, if people are careless or even malicious about using data, you can take steps to make that stop. If there's no policy, or if it's out of date, of course create a new one. And if you have tools that can alert you to changes before they crash data feeds or blow up dashboards, you'd be negligent not to try to use them! Our Data Cookbook solution is an affordable and lightweight tool that would help with these issues, and we'd be happy to demonstrate it to you.
Ultimately, however, what helps resolve this situation, and countless others like it, is recognizing that data is an organizational asset, not one owned by a particular person, department, unit, or line of business. Data stewardship is performed on behalf of the organization, which includes not only management and the steward's domain but every other domain as well. Data stewards and subject matter experts are not infallible, nor should anyone expect them to be. But their time at the office is limited, and their availability to handle questions even more so. It's not unreasonable to expect users to consult a knowledge base or other resources prior to pointing fingers, throwing up their hands, or making other helpless, angry gestures. And it's not unreasonable to expect users to ask questions, make out suggestions, or even point out errors in a respectful, collaborative, non-judgmental way.
When we agree that data is an important asset, we handle it carefully. We pay it significant attention. We look for ways to extract value from it, which includes making it available to our colleagues, making sure it's up to date, regularly reviewing its relevance and quality, and perhaps most importantly being open to suggestion about ways to improve it!
We opened this post by mentioning reference data, and we also brought master data and then data stewardship in. That's a lot to cover in a few hundred words, and we haven't really done justice to any of those topics. But we hope we've given you some tools to open up the conversation at your workplace about working together to maximize data's value, and about taking easy steps to improve things across the board, even if your area in particular seems to see less benefit. We've been having this conversation for better than two decades at this point. If it were a person it would be old enough to drink a beer! And we'd be happy to hoist a virtual pint with you to help you further this and similar conversations closer to home.
We hope you found this blog post useful. Also check our data governance spotlight resources located at https://www.datacookbook.com/spotlights. IData has a solution, the Data Cookbook, that can aid the employees and the organization in their data governance, data intelligence, data stewardship, artificial intelligence, and data quality initiatives. IData also has experts that can assist with data governance, reporting, integration and other technology services on an as-needed basis. Feel free to contact us and let us know how we can assist.
Image Credit: StockSnap_HWEMM8GQV4_ThinLine_BP #1335
