Data Quality
Definition
Data quality is the degree to which data is fit for use. Poor (“dirty”) data — inconsistent formats, missing values, errors, duplicates — undermines analytics and decisions, so it is profiled, cleansed, and monitored, especially during ETL into a warehouse.
Core Ideas
Dirty data
Common problems:
- Inconsistent standards in structure, format, or values (full name vs first/last; mixed currencies)
- Missing data / absent default values (every field should have a sensible default)
- Spelling errors or data in the wrong fields
- Duplicate records (e.g. two “rulers”; check by name, confirm with the client)
- Noisy data — random error; sometimes best spotted by plotting
- Outliers — values above or below 3 × standard deviation
- Out-of-range, non-standard, invalid, differing cultural rules, varying formats, cosmetic issues
Dimensions of quality
- Accuracy — matches the true value (if every pen is $10, anything else is inaccurate;
0000000is not a real phone number) - Validity — within a proper range (a price of 0 is invalid)
- Consistency — represented uniformly (whole name vs first + last)
- Completeness — no nulls or missing values
- Uniformity — standardized representation (strip all dashes from phone numbers)
The data-quality assurance process
- Checking — profile the data to discover inconsistencies and anomalies
- Logging — record issues (a DQ log table in the warehouse:
LogID, RowID, DBName, TableName, RuleNo, Actionwhere Action ∈ Reject / Fix / Allow) - Correcting — perform data cleaning
During ETL this includes statistical analysis (evaluate new data against history for outliers using mean, median, mode, standard deviation, range), integrity validation, standardization (via transformation rules), and tracking (a metadata repository). Data assurance is pre-processing; data validation for the warehouse is post-processing.
Relationships
- Data Warehousing and Business Intelligence — where profiling and cleansing live in the ETL pipeline
- Dimensional Modeling — null handling and conformity in dimensions
- Data Science Methodology — data preparation consumes 70–90% of project effort
- Data Leakage — a distinct data hazard affecting model validity