Study Guide
Module 7 · 6 parts · Lecture 7 deck + the class notebook📌 Parts 1–4 come from the Lecture 7 deck. Parts 5–6 walk through the class notebook (
Social_Network_Ads.csv), whose outputs were reproduced to confirm every number. The notebook has no written explanations, so code behavior there comes from the scikit-learn / NumPy / Matplotlib documentation (links on the Notes tab). All diagrams on this page are redrawn for study. Purple Study aid boxes are memory tricks, OOP reminders, extra runs on the class data, and links to earlier modules — not lecture content.🧭 Part 1 — What K-NN Is & How It Works
What it is
- K-Nearest Neighbors (K-NN) — a simple, intuitive supervised learning algorithm used for both classification and regression.
- How it predicts — a new point's outcome is based on the outcomes of its K closest neighbors.
- Why it's taught — one of the simplest algorithms in machine learning, and a great way to build intuition for classification.
🧩 Study aid — "supervised" (Module 1). Supervised learning learns from labeled data, where the correct answer is already known. K-NN needs those labels: they're what the neighbors vote with.
"Lazy learning"
- No explicit model — K-NN doesn't build a model during training; it just stores the training data.
- Instance-based — every prediction is made directly from the stored examples ("instances").
- The work moves to prediction time — each new point has to be compared with the stored data, which is why prediction can be slow on large datasets (Part 4).
The four steps
| Step | What happens | In the picture above |
|---|---|---|
| 1 · Choose K | Pick the number of neighbors to look at. | K = 3 |
| 2 · Calculate distances | Measure the distance from the new point to every point in the training data. | Star → every shape |
| 3 · Find the K closest | Keep the K points with the smallest distances — the "nearest neighbors." | The 3 shapes inside the dashed circle |
| 4 · Vote (or average) | Classification: majority vote of the neighbors' classes. Regression: average of their values. | 2 circles vs 1 triangle → Class 0 |
The lecture sums it up in three stages: store the data (keep all training examples and their labels) → find the K nearest neighbors → majority vote (predict the class with the most neighbors).
⚠️ Internal inconsistency in the lecture: one of its K = 3 pictures also circles a green square inside the neighborhood, even though the same picture's tally says 0 green neighbors. Go by the tally and the rule: with K = 3, only the 3 closest points vote.
Classification vs. regression
| Classification | Regression | |
|---|---|---|
| Predicts | A class label | A continuous value |
| Final step | Majority vote among the K nearest neighbors | Average of the K nearest neighbors' values |
| Finding neighbors | Exactly the same in both — only the final step changes. | |
📌 Practice numbers: the labels and values below are made up to show the two final steps. Only the rule (vote vs. average) comes from the lecture.
🎚️ Part 2 — Choosing K
| Small K (e.g., K = 1) | Large K (e.g., K = 15) | |
|---|---|---|
| Boundary | Jagged; bends around single points | Smoother |
| Noise & outliers | Sensitive to noise; easily swayed by outliers | Less swayed — many neighbors vote |
| Risk | Overfits to noise | Can blur the boundary between classes; may underfit |
- Common practice — try several values of K and use cross-validation to pick the best one.
- Odd K — in binary classification, K is usually an odd number to avoid tie votes (with K = 4, a 2–2 split has no winner).
🧩 Study aid — terms from Module 5. Overfitting = the model memorizes the training data, noise included, and does worse on new data. Underfitting = the model is too simple to capture the pattern. Cross-validation = testing a model more reliably than a single train/test split: k-fold splits the data into k parts, trains on k − 1, tests on the last, and repeats so every part gets a turn as the test set.
⚠️ Name trap: the k in "k-fold cross-validation" (how many parts the data is split into) is not the K in K-NN (how many neighbors vote). You can use 5-fold cross-validation to compare K = 3, 5, 7 …
💡 Study aid — K on the class data (not run in class). Same split and scaling as the notebook, only K changed:
K = 1 scores a perfect 1.00 on the training data (every point's nearest neighbor is itself) but drops to 0.87 on the test data — overfitting in action. The class value, K = 5, scores 0.93 on test. Not on the quiz.
| K | Training accuracy | Test accuracy |
|---|---|---|
| 1 | 1.00 | 0.87 |
| 3 | 0.91 | 0.93 |
| 5 (class) | 0.91 | 0.93 |
| 7 | 0.91 | 0.93 |
| 15 | 0.91 | 0.93 |
| 25 | 0.90 | 0.93 |
📏 Part 3 — Distance Metrics & Feature Scaling
Distance: how K-NN decides who's a neighbor
| Metric | Formula | What it measures |
|---|---|---|
| Euclidean | d(p,q) = √( Σ (pᵢ − qᵢ)² ) | Straight-line distance — the most common choice for K-NN. |
| Manhattan | d(p,q) = Σ |pᵢ − qᵢ| | Sums the absolute differences — like navigating a city grid. |
- pᵢ and qᵢ — the two points' values for feature i; Σ adds up over every feature.
- Why it matters — different distance metrics can lead to different neighbors, and therefore different predictions.
📌 Practice numbers: the points above and below are made up. Only the formulas come from the lecture.
| Points | Differences | Euclidean | Manhattan |
|---|---|---|---|
| (1, 1) → (4, 5) | 3, 4 | √(9 + 16) = 5 | 3 + 4 = 7 |
| (0, 0) → (6, 8) | 6, 8 | √(36 + 64) = 10 | 6 + 8 = 14 |
| (2, 7) → (2, 3) | 0, 4 | √(0 + 16) = 4 | 0 + 4 = 4 |
| (1, 2, 3) → (3, 4, 4) | 2, 2, 1 | √(4 + 4 + 1) = 3 | 2 + 2 + 1 = 5 |
💡 Study aid — memory trick. Euclidean = as the crow flies (square, add, square-root). Manhattan = as the taxi drives (just add the gaps). Manhattan is never shorter than Euclidean; they're equal only when the points differ in a single feature (third row).
Why feature scaling matters for K-NN
- K-NN relies entirely on distance calculations.
- Big scales dominate — a feature on a larger scale (e.g., income in the thousands) dominates the distance, drowning out smaller-scale features (e.g., age).
- The fix — always apply normalization (min-max) or standardization (z-score) before using K-NN, so both features contribute equally.
- Common trap — forgetting to scale is one of the most common beginner mistakes with this algorithm.
The class data shows it. Same new person, two possible neighbors:
- Unscaled — an 11-year age gap adds just 11 to the distance, but a $2,000 salary gap adds 2,000. Neighbor A (11 years older) "wins," even though B is the same age and earns almost the same.
- Scaled — each gap is divided by that feature's spread (training std: age ≈ 10.1, salary ≈ 34,491). Now the age gap ≈ 1.09 and the salary gap ≈ 0.06, so B is correctly the nearer neighbor.
🧩 Study aid — the two scaling formulas (Module 2). Min-max normalization:
x' = (x − min) / (max − min) → squeezes values into [0, 1]; sensitive to outliers. Z-score standardization: z = (x − μ) / σ → mean 0, standard deviation 1; more robust to outliers. The class notebook uses z-score (StandardScaler).💡 Study aid — K-NN without scaling (not run in class). Training the same K = 5 model on the raw ages and salaries drops test accuracy from 0.93 to 0.83 (confusion matrix
[[59 9] [8 24]]). For the 30-year-old earning $87,000, the five raw-data neighbors all earn $86,000–$88,000 but range in age from 18 to 41 — age barely counts.⚖️ Part 4 — Strengths, Weaknesses, Uses & Summary
Strengths vs. weaknesses
| Strengths | Weaknesses |
|---|---|
| Simple and intuitive to understand | Slow at prediction time on large datasets |
| No real training phase | Sensitive to irrelevant features and unscaled data |
| Works for both classification and regression | Struggles with high-dimensional data |
| Naturally handles multi-class problems | Memory-intensive — it stores the entire dataset |
💡 Study aid — connect the weaknesses to "lazy." Storing the whole dataset → memory-intensive. Skipping training → all the distance work happens at prediction time → slow predictions. Distance adds up a term for every feature, useful or not → irrelevant and unscaled features distort who counts as a neighbor. Two strengths come from the same place: no training phase, and a majority vote works with any number of classes (multi-class).
⚠️ Not explained in the source: why K-NN struggles with high-dimensional data (data with many features). The lecture states it without giving a reason.
Real-world use cases
| Use | How "nearest neighbors" shows up |
|---|---|
| Recommendations | Find users or items similar to a given one. |
| Image recognition | Classify an image based on similar-looking labeled images. |
| Anomaly detection | Flag points that have very few close neighbors. |
| Medical diagnosis | Classify a patient based on similar historical patient records. |
Summary & key takeaways
- K-NN predicts based on the majority vote (or average) of the K closest data points.
- There's no real "training" — it's a lazy, instance-based learner.
- Choosing K and scaling your features are the two most important decisions.
- A simple, intuitive baseline for both classification and regression tasks.
🐍 Part 5 — Class Notebook: Load, Split, Scale, Train & Predict
The whole notebook on one line
import→
load CSV → X, y→
split 75 / 25→
scale→
K-NN .fit (store)→
predict (vote)→
confusion matrix + accuracy→
plot train & test
🧩 Study aid — this is the Module 6 notebook with the model swapped. Same dataset, same split, same scaling, same evaluation and plots. The only code that changes is the two lines that build the classifier (step 5), plus the plot titles and grid step. Part 6 compares the results side by side.
1 · Import the libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
- np — arrays and math (
concatenate,meshgrid,arange). - plt — plotting the decision boundary.
- pd — loading the CSV into a DataFrame.
2 · Import the dataset → features X, target y
dataset = pd.read_csv('Social_Network_Ads.csv')
X = dataset.iloc[:, :-1].values # features: Age, EstimatedSalary
y = dataset.iloc[:, -1].values # target: Purchased (0 or 1)
| Age | EstimatedSalary | Purchased |
|---|---|---|
| 19 | 19000 | 0 |
| 35 | 20000 | 0 |
| 26 | 43000 | 0 |
| … | … | … |
| 400 rows · 257 with 0, 143 with 1 · ages 18–60 · salaries 15,000–150,000 (counted from the class CSV) | ||
iloc[:, :-1]— all rows, every column except the last → the features.iloc[:, -1]— all rows, last column only → the target..values— hands back the plain NumPy array.
🔤 X / y reminder. Uppercase X = the feature matrix (2-D: 400 rows × 2 columns). Lowercase y = the target vector (1-D: 400 values). The casing carries through:
X_train, X_test, y_train, y_test, y_pred.3 · Split into training and test sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25, random_state = 0)
test_size = 0.25— 100 test rows, 300 training rows.- Return order —
X_train, X_test, y_train, y_test. random_state = 0— same shuffle every run (and the same split as Module 6).- Class balance — y_train: 189 zeros, 111 ones · y_test: 68 zeros, 32 ones.
4 · Feature scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train) # learn mean & std from TRAIN, then scale
X_test = sc.transform(X_test) # reuse the TRAIN mean & std
| Age | EstimatedSalary | |
|---|---|---|
Training mean (learned by fit) | ≈ 38.1 | ≈ 69,583 |
Training std (learned by fit) | ≈ 10.1 | ≈ 34,491 |
| First training row, before | 44 | 39000 |
| Same row, after | 0.58164944 | −0.88670699 |
- Fit on train only — the scaler learns from training data and reuses those numbers on test data, so no test information leaks in.
- Extra important here — K-NN is pure distance, so unscaled salary would drown out age (Part 3).
📌 The mean and std values aren't printed in the notebook; they were read from the fitted scaler (
sc.mean_, sc.scale_) on the same split.🧩 OOP reminder — the scaler remembers.
sc = StandardScaler() creates an object (like new StandardScaler() in Java). fit_transform stores the training mean and std inside sc as fields. Every later sc.transform(...) — on X_test, on the new person, on the plot grid — reuses them.5 · Create and "train" the K-NN classifier
from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors = 5, metric = 'minkowski', p = 2) # K = 5, Euclidean
classifier.fit(X_train, y_train) # store the training data
KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
metric_params=None, n_jobs=None, n_neighbors=5, p=2,
weights='uniform')
| Argument | Meaning | Lecture link |
|---|---|---|
n_neighbors = 5 | K — how many neighbors vote. | Step 1 · odd K avoids ties |
metric = 'minkowski' | The distance formula family. | Distance metrics |
p = 2 | With Minkowski, p = 2 is Euclidean; p = 1 is Manhattan. | Euclidean = most common |
weights='uniform' (default, printed) | Every neighbor's vote counts equally. | Majority vote |
- What
.fitdoes here — K-NN is a lazy learner, so fitting just stores the scaled training points and their labels. No weights or bias are learned (compare Module 6). - The printed line — Jupyter echoes the model's settings, defaults included; it isn't a result.
algorithm,leaf_sizeandn_jobscontrol how scikit-learn searches for neighbors and aren't covered in the lecture.
📌 From the scikit-learn docs: Minkowski distance is
(Σ |pᵢ − qᵢ|p)1/p. With p = 1 that's the lecture's Manhattan formula; with p = 2 it's the Euclidean formula. The lecture itself never names Minkowski — it only appears in the notebook.🧩 OOP reminder — same method name, different behavior. Both
LogisticRegression (Module 6) and KNeighborsClassifier have .fit() and .predict(). That's like two Java classes implementing the same interface: you call the same methods, but each class does its own thing inside. Logistic regression's fit learns weights; K-NN's fit just saves the data in the object's fields.6 · Predict a single new result
print(classifier.predict(sc.transform([[30, 87000]])))
[0]
- Inside-out —
[[30, 87000]]→sc.transformscales it →classifier.predictruns the K-NN vote →print. - Double brackets — one row, two feature values: 2-D, shaped like X.
- Result —
[0]: a 30-year-old earning $87,000 is predicted not to purchase.
What the vote looked like (reproduced with classifier.kneighbors, not run in class):
💡 Study aid — the vote as a probability.
classifier.predict_proba(...) on this person returns [[1. 0.]]: 5 of 5 neighbors are class 0, so "probability" of class 0 = 5/5 = 1.0. For K-NN that number is just the share of neighbor votes. Not on the quiz.📊 Part 6 — Class Notebook: Evaluate, Plot & Compare with Logistic Regression
7 · Predict the test set results
y_pred = classifier.predict(X_test)
print(np.concatenate((y_pred.reshape(len(y_pred),1), y_test.reshape(len(y_test),1)),1))
[[0 0]
[0 0]
…
[1 0] # predicted 1, actually 0 → false positive
…
[0 1] # predicted 0, actually 1 → false negative
…]
- Left column = predicted (
y_pred), right column = actual (y_test). reshape(len(y_pred), 1)turns each flat array into a column;np.concatenate(..., 1)glues the columns side by side.- Behind the scenes — for each of the 100 test people, K-NN finds the 5 nearest of the 300 stored training points and takes a vote. Nothing was "learned" ahead of time.
8 · Confusion matrix and accuracy
from sklearn.metrics import confusion_matrix, accuracy_score
cm = confusion_matrix(y_test, y_pred) # actual first, predicted second
print(cm)
accuracy_score(y_test, y_pred)
[[64 4]
[ 3 29]]
0.93
| Predicted 0 | Predicted 1 | |
|---|---|---|
| Actual 0 | TN = 64 correctly said "won't buy" | FP = 4 said "buy" — didn't |
| Actual 1 | FN = 3 said "won't buy" — did | TP = 29 correctly said "buy" |
- Accuracy — (TP + TN) / total = (29 + 64) / 100 = 0.93.
- Mistakes — 4 + 3 = 7 of 100.
- Row totals — 64 + 4 = 68 actual 0s; 3 + 29 = 32 actual 1s (matches y_test).
⚠️ Layout reminder (same trap as Module 6). scikit-learn prints class 0 first:
[[TN FP] [FN TP]]. Module 5's matrix put the positive class first. The layout comes from the scikit-learn documentation; the notebook doesn't label it.💡 Study aid — Module 5 formulas on this result (not computed in class). Precision = TP / (TP + FP) = 29 / 33 ≈ 0.88. Recall = TP / (TP + FN) = 29 / 32 ≈ 0.91. F1 = 2·P·R / (P + R) ≈ 0.89.
9 · Visualize the training and test set results
from matplotlib.colors import ListedColormap
X_set, y_set = sc.inverse_transform(X_train), y_train # A · back to real units (X_test, y_test for the test plot)
X1, X2 = np.meshgrid(np.arange(start = X_set[:, 0].min() - 10, stop = X_set[:, 0].max() + 10, step = 1),
np.arange(start = X_set[:, 1].min() - 1000, stop = X_set[:, 1].max() + 1000, step = 1)) # B · grid
plt.contourf(X1, X2, classifier.predict(sc.transform(np.array([X1.ravel(), X2.ravel()]).T)).reshape(X1.shape),
alpha = 0.75, cmap = ListedColormap(('red', 'green'))) # C · color regions
plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())
for i, j in enumerate(np.unique(y_set)):
plt.scatter(X_set[y_set == j, 0], X_set[y_set == j, 1], c = ListedColormap(('red', 'green'))(i), label = j) # D · dots
plt.title('K-NN (Training set)') # 'K-NN (Test set)' in the second cell
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()
| Piece | What it does |
|---|---|
A · sc.inverse_transform(...) | Undoes the scaling so the axes show real ages and salaries. |
B · np.meshgrid(...), step = 1 | Builds a grid of (age, salary) points: 10 years past the age range and 1,000 past the salary range, in steps of 1. That's 62 ages × 137,000 salaries ≈ 8.5 million points. |
C · plt.contourf(...) | Flattens the grid, scales it, predicts every point with K-NN, folds the answers back into the grid shape, and paints the background: red = predicted 0, green = predicted 1. |
D · plt.scatter loop | Plots each real point in its true color: red for 0, green for 1. |
- Boundary shape — irregular and jagged, with small pockets. K-NN follows whatever the local neighbors say, so the edge bends around clusters of points instead of being a straight line.
- Pattern — younger, lower-salary people fall in red (didn't purchase); older people and high earners fall in green.
- Misclassified points — a dot on the wrong color. On the test plot those are the 4 false positives (red dots on green) and 3 false negatives (green dots on red).
- Two plots this time — the Module 6 notebook only plotted the training set; this one plots both.
- The warning above each chart — the same harmless Matplotlib
c-argument warning as Module 6.
💡 Study aid — "slow at prediction time," live. Each plot asks K-NN for about 8.5 million predictions, and every one needs its 5 nearest neighbors. In the notebook's saved run, each plotting cell finished about 3½ minutes after the one before it — the lecture's first weakness, in real life.
10 · Logistic regression (Module 6) vs. K-NN (Module 7) — same data
| Logistic regression | K-NN (K = 5) | |
|---|---|---|
| Model line | LogisticRegression(random_state = 0) | KNeighborsClassifier(n_neighbors = 5, metric = 'minkowski', p = 2) |
What .fit does | Learns weights and a bias | Stores the training data (lazy) |
| Boundary | Straight line | Irregular, jagged |
| Confusion matrix | [[65 3] [8 24]] | [[64 4] [3 29]] |
| Accuracy | 0.89 | 0.93 |
| Missed buyers (FN) | 8 | 3 |
| Recall (study aid) | 0.75 | ≈ 0.91 |
| New person (30, $87,000) | 0 | 0 |
- Why K-NN did better here — the purchase pattern isn't a straight line, and K-NN's flexible boundary can follow it. It traded one extra false positive (4 vs 3) for five fewer missed buyers (3 vs 8).
📌 The logistic regression numbers come from the Module 6 class notebook (same CSV, same split, same scaling). The lecture doesn't compare the two models; this table is a study aid.