Explanation 1
The live starter code is self-contained and uses only runner-compatible functions.
Loading
Lesson 5 of 22
Treat cleaning as an auditable chain of schema, type, missingness, range, and key checks.
Motivation
A model cannot repair an incorrect unit, duplicated key, or impossible value. Validation should fail near the cause, before plausible-looking output is produced.
Why this matters
Treat cleaning as an auditable chain of schema, type, missingness, range, and key checks.
Packages and data
readr, readxl, dplyr, and janitor are recommended; the live example stays in base R.
Explanation 1
The live starter code is self-contained and uses only runner-compatible functions.
Explanation 2
Book-only outputs are labelled when the separate source dataset or advanced package was not supplied.
Explanation 3
R code, paths, function names, and formulas remain left-to-right in every locale.
Terminology
Expected columns, types, and structural rules.
date must be unique and ordered
NA can mean unavailable, inapplicable, suppressed, or failed collection.
Count by row, column, and time
A value deserving investigation, not automatic deletion.
Compare with domain rules and source
Notation and formulas
validation = schema + keys + ranges + missingness + provenance
Validation turns substantive expectations into executable checks.
Worked example
Scenario
How should a deliberately impossible 99% rate and three missing inflation cells be handled?
R check
dirty <- data.frame(date=as.Date('2026-01-01')+0:4, inflation=c(2.1,NA,2.8,NA,3.0), rate=c(4.5,4.5,99,4.25,4.25))
print(colSums(is.na(dirty)))
clean <- dirty[!is.na(dirty$inflation), ]
clean$rate[clean$rate > 20] <- NA_real_
print(clean)
stopifnot(!anyDuplicated(clean$date))Replacing a suspect value with missingness is safer than inventing a correction. Limitation: The threshold above 20% is a domain rule for this teaching context, not a universal law.
Visual
The constructed table contains two missing inflation cells and one impossible rate.
R connection
Run the self-contained starter code in the protected STATLAB R runner. The code prints an auditable result and avoids network or unrestricted file access.
Live R Lab
Run a self-contained example, verify its output, and explain one limitation for importing, cleaning, and validating data.
Ready to run
Common mistake
Do not remove an outlier because it makes a model fit better.
STATLAB Tip
Keep a data dictionary with name, type, unit, definition, transformation, and source.
Guided practice
Scenario
Insert a duplicate date and write code that reports the duplicate rows without deleting them.
Compare your result with the definition, units, and model assumptions—not only with a target number.
Exercises
Complete these without looking at the selected solutions. More than one defensible program may exist.
Explanation 1
Check five required column names.
Explanation 2
Count missing values by row and column.
Explanation 3
Propose and justify an exchange-rate range rule.
Selected solutions
These are compact solution routes. Confirm dimensions, units, and any changed modelling choices.
Explanation 1
stopifnot(setequal(names(data), required))
Explanation 2
rowSums(is.na(data)); colSums(is.na(data))
Explanation 3
Use a wide, documented domain range and investigate—not silently delete—violations.
Chapter summary
Treat cleaning as an auditable chain of schema, type, missingness, range, and key checks.
Explanation 1
Inspect inferred types.
Explanation 2
Preserve raw files.
Explanation 3
Interpret missingness and outliers.
Explanation 4
Validate schema, keys, ranges, and counts.
Terminology
CSV import
Use in Importing, Cleaning, and Validating Data.
missingness
Use in Importing, Cleaning, and Validating Data.
column counts
Use in Importing, Cleaning, and Validating Data.
validation gate
Use in Importing, Cleaning, and Validating Data.
References and provenance
Safavi (2026), Chapter 5. Student notes: Mohammad Safavi, Ph.D., STATLAB Academy, Version 1.0.
Resource
The authoritative 125-page English PDF accompanies this native lesson.
DownloadReflection
Name the assumption, evidence you would seek, and how the recommendation might change.
Exit check
Reach 70% to complete the chapter. Explanations appear after submission.
Checkpoint
Question 1 of 2. Answered 0/2. Passing score: 70%.