← All courses CSIS 3352 · Module 3

Hands-on LAB: Data Processing

📅 Assignment 1 (100 pts) — due 9/8 · Reviewed on Midterm (Modules 1–9) — moved to Tue Oct 27
📌 The Module 3 lab source is a Python notebook (data_preprocessing_tools.ipynb) with code only — no written learning objectives or prose. The objectives, definitions, and rubric points below are derived from the script's own structure and the official pandas / scikit-learn documentation, and are flagged as externally sourced rather than taken from provided course prose.
⚠️ Source conflict: the worked notebook uses random_state = 1; the separate template notebook uses random_state = 0. This page treats 1 as canonical because it generated the printed outputs shown in the worked file. Confirm against Dr. Amin's answer key.

Study Guide

Module 3 · The 7-stage preprocessing pipeline
📌 Definitions below are sourced from the official pandas / scikit-learn documentation (the lab file itself contains no prose). Confidence: high for library behavior; see the Notes tab for source links.
import numpy as np import matplotlib.pyplot as plt import pandas as pd

Terms

  • numpy (np) — numerical computing library; arrays, np.nan, np.array().
  • matplotlib.pyplot (plt) — plotting; part of the standard import block by convention even though it is unused in this script.
  • pandas (pd) — data loading / DataFrames; provides pd.read_csv().
dataset = pd.read_csv('Data.csv') X = dataset.iloc[:, :-1].values y = dataset.iloc[:, -1].values print(X) print(y)

Terms

  • read_csv — loads the CSV into a DataFrame. The dataset has 4 columns: Country, Age, Salary, Purchased (10 rows).
  • iloc[:, :-1] — integer-location slice: all rows, all columns except the last → features (X).
  • iloc[:, -1] — all rows, last column only → dependent variable (y).
  • .values — returns the underlying NumPy array from the pandas object.
  • Missing data present — one Age and one Salary value are nan in the raw data (handled in Stage 3).
from sklearn.impute import SimpleImputer imputer = SimpleImputer(missing_values=np.nan, strategy='mean') imputer.fit(X[:, 1:3]) X[:, 1:3] = imputer.transform(X[:, 1:3]) print(X)

Terms

  • SimpleImputer — from sklearn.impute; fills missing values.
  • missing_values=np.nan — which value counts as "missing".
  • strategy='mean' — replace with the column mean (other options: median, most_frequent, constant).
  • X[:, 1:3] — Age & Salary columns only (column 0, Country, is text).
  • fit then transform — fit learns the means; transform writes them back into the array.
from sklearn.compose import ColumnTransformer from sklearn.preprocessing import OneHotEncoder ct = ColumnTransformer(transformers=[('encoder', OneHotEncoder(), [0])], remainder='passthrough') X = np.array(ct.fit_transform(X)) print(X)

Terms

  • ColumnTransformer — from sklearn.compose; applies a transformer to selected columns.
  • OneHotEncoder — from sklearn.preprocessing; turns a category column into binary dummy columns.
  • transformers=[('encoder', OneHotEncoder(), [0])] — name, transformer, target column list.
  • remainder='passthrough' — keep untouched columns; the default 'drop' would delete them.
  • Result — three dummy columns (France, Germany, Spain) move to the front; Age/Salary become columns 3 & 4.
from sklearn.preprocessing import LabelEncoder le = LabelEncoder() y = le.fit_transform(y) print(y)

Terms

  • LabelEncoder — from sklearn.preprocessing; encodes labels as integers.
  • fit_transform(y) — learns the label set and converts in one call.
  • Result — No → 0, Yes → 1 (assigned alphabetically).
from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1) print(X_train) print(X_test) print(y_train) print(y_test)
⚠️ Source conflict — worked file: random_state = 1; template file: random_state = 0. Canonical here = 1 (it produced the notebook's printed splits). Verify against the answer key.

Terms

  • train_test_split — from sklearn.model_selection.
  • test_size = 0.2 — 20% test / 80% train (8 train rows, 2 test rows here).
  • random_state = 1 — seeds the shuffle so the split is reproducible.
  • Return order — X_train, X_test, y_train, y_test.
from sklearn.preprocessing import StandardScaler sc = StandardScaler() X_train[:, 3:] = sc.fit_transform(X_train[:, 3:]) X_test[:, 3:] = sc.transform(X_test[:, 3:]) print(X_train) print(X_test)

Terms

  • StandardScaler — from sklearn.preprocessing; standardizes to mean 0, unit variance.
  • [:, 3:] — scale Age/Salary only, not the 0/1 dummy columns.
  • fit_transform(X_train) vs transform(X_test) — fit on train only, then apply to test (avoids data leakage).
  • Order — scaling is done after the split, not before.

Flashcards

Question
Answer

Practice Quiz

Concept Check

Explain each step in your own words, then self-grade against the rubric

Code Lab

Reproduce the preprocessing script from memory. Pick a single stage to drill, or Full Program for the whole thing. Each non-blank line is scored pass / fail against the canonical version — spacing and quote style are ignored, but token content and capitalization must match (Python is case-sensitive). The program does not need to run; the score is simply how many lines you reproduced.

① Warm-up — put the stages in order
② Reproduce the code

Notes Template

Module 3 · Print this tab for handwritten recall practice

📚 Reference documentation (official)

The 7 stages (in order)

Library → module map

fit vs transform vs fit_transform

One-hot vs label encoding

Why scale only [:, 3:]

Why fit scaler on train only