Design Tip #117 Dealing with Data Quality: Don’t Just Sit There, Do Something!
Design Tip #117 Dealing with Data Quality: Don’t Just Sit There, Do Something! Most data quality problems can be traced back to the data capture systems because, historically, they have only been responsible for the level of data quality needed to support transactions. What works for transactions often won’t work for analytics. In fact, many of the attributes we need for analytics are not even necessary for the transactions, and therefore capturing them correctly is just extra work. By requiring better data quality as we move forward, we are requiring the data capture system to meet the needs of both transactions and analytics. Changing the data capture systems to get better data quality is a long term organizational change process. This political journey is often paralyzing for those of us who didn’t expect to be business process engineers in addition to being data warehouse engineers! Do not let this discourage you. You can take some small, productive steps in the short term t