State why data quality matters and describe the six common data quality dimensions.

• Why it matters — bad data leads to bad decisions, broken models, and wasted time ("garbage in, garbage out"); poor data quality is estimated to cost organizations millions of dollars every year, and cleaning is usually the single largest time investment in a real data science project.
• Accuracy & Completeness — Accuracy = data correctly reflects real-world values and events; Completeness = all required data is present and nothing important is missing.
• Consistency & Timeliness — Consistency = data is uniform across systems and formats ("USA" / "U.S." / "United States"); Timeliness = data is up-to-date and available when it is needed.
• Uniqueness & Validity — Uniqueness = each piece of data is unique and not duplicated unnecessarily; Validity = data conforms to defined rules, formats, and business constraints (a birth date is a valid date).

Define missing values, give their common causes, and name the three types of missingness.

• Definition — missing values are empty or null cells where data should exist.
• Common causes — a skipped form field, sensor failure, a merge between mismatched sources, or a system error.
• Three types — Missing Completely at Random (MCAR), Missing at Random (MAR), and Missing Not at Random (MNAR).

Describe the methods for handling missing values and what should guide the choice.

• Deletion — drop rows/columns with too many missing values, but use it carefully because you lose data.
• Imputation — fill in a statistically reasonable replacement using the mean, median, or mode.
• Forward-fill / backward-fill — carry the nearest known value forward or backward; common in time series.
• Cause first — understand why the value is missing before deciding how to handle it; the cause should drive the fix.

Explain what duplicate records are, why they matter, and how to detect and remove them.

• What & why — the same real-world entity appears more than once; common causes are repeated form submissions, merging multiple sources, or system syncing errors, and duplicates can silently bias analysis and inflate counts, averages, and totals.
• Exact vs. near-duplicates — exact duplicates are identical across every column (easy to drop); near-duplicates differ slightly (typos, spacing, formatting) and often need fuzzy matching.
• Tools & caution — in pandas, .duplicated() finds them and .drop_duplicates() removes them; always double-check what actually counts as a duplicate before removing anything.

Define an outlier and explain how to detect one using the IQR and Z-score methods.

• Definition — a data point that differs significantly from the rest of the dataset; it can be a genuine rare event or an error (e.g., one $500,000 order where the average is $50).
• IQR method — flag x if x < Q1 − 1.5×IQR or x > Q3 + 1.5×IQR, where IQR = Q3 − Q1; more robust to extreme values.
• Z-score method — z = (x − μ) / σ; commonly flag |z| > 3; assumes a roughly normal distribution.
• Visual methods — box plots and scatter plots make outliers easy to spot by eye.

List and explain the options for handling an outlier once it is found.

• Investigate first — decide whether it is a genuine extreme value or a data-entry / sensor error.
• Remove or cap — remove it if it is clearly an error that would distort the analysis, or cap/limit it (winsorizing) by pulling extremes toward a reasonable boundary.
• Transform or keep — transform the data (e.g., a log transform) to reduce the influence of extremes, or keep it if it is a genuine, meaningful data point for your question.

Explain why categorical data must be encoded and why the choice of method matters.

• Why encode — machine learning models need numbers, not text labels, so categories like "Red", "Blue", "Green" must be converted to a numeric form.
• Why the method matters — a poor encoding choice can introduce a false sense of order or relationship between categories.

Compare label encoding and one-hot encoding, including when to use each.

• Label encoding — assigns each category an integer (Red=0, Blue=1, Green=2); simple, but implies a false order, so it is best for ordinal data (e.g., Low/Medium/High).
• One-hot encoding — creates a separate binary (0/1) column per category; no false order but adds more columns, so it is best for nominal data (no natural order).

Explain why features are scaled and where scaling matters most.

• The problem — features on very different scales (e.g., Age 0–100 vs. Income 0–200,000) can distort many ML models.
• The fix — scaling puts all features on a comparable range so no single feature unfairly dominates.
• Where it matters — especially for distance-based models (k-NN, k-means) and gradient-based models (neural networks).

Give the min-max normalization and z-score standardization formulas and explain when to use each.

• Min-max normalization — x' = (x − min(x)) / (max(x) − min(x)); rescales into a fixed [0, 1] range but is sensitive to outliers.
• Z-score standardization — z = (x − μ) / σ; centers data at mean 0 with standard deviation 1 and is more robust when outliers are present.
• When to use each — use min-max when you need a strictly bounded range (e.g., neural network inputs); use standardization when data may contain outliers or is not uniformly distributed.

Define incorrect data, give examples, and explain what is needed to catch it.

• Definition — data that is present but wrong: typos, wrong units, or impossible values.
• Examples — age = 250, a negative price, "Februaryy" instead of "February", and inconsistent date formats.
• Catching it — requires domain knowledge and clear validation rules.

Define noise, distinguish it from outliers, and explain where it appears and why it matters.

• Definition — noise is random variation or fluctuation in data that is not meaningful signal.
• Vs. outliers — noise is small and everywhere; outliers are extreme and isolated.
• Where & why — common in sensor data, audio/image data, and survey responses; too much noise can hide the real underlying pattern.

Define data validation, list the four checks, and state when validation should happen.

• Definition — checking that data meets defined rules before it is used.
• Four checks — type check (is it the expected number/date/text?), range check (e.g., age 0–120, percentage 0–100), format check (email, phone, or date properly formatted), and consistency check (e.g., end date after start date).
• When — as early as possible, ideally right at the point of collection.

Define data integrity and explain the practices that maintain it.

• Definition — the overall accuracy, consistency, and reliability of data across its entire lifecycle.
• Ongoing, not one-time — it is an ongoing responsibility, not a one-time cleaning step.
• Practices — access controls, audit trails, backups, validation pipelines, and clear data governance.