Classification

machine-learning

supervised-learning

classification

Last week you learned linear regression, which predicts a number. This week introduces classification, where the output can take on only one of a small…

Published

July 7, 2026

Last week you learned linear regression, which predicts a number. This week introduces classification, where the output \(y\) can take on only one of a small handful of possible values instead of any number in an infinite range.

Linear regression is not a good algorithm for classification problems. Understanding why leads directly to logistic regression, one of the most widely used learning algorithms today.

Review Questions

1. What is the main difference between regression and classification outputs?

Answer

Regression predicts a number from a large or infinite set of values. Classification predicts a category from a small, finite set of labels.

Classification Examples

Recall examples from supervised learning:

Spam filtering: is this email spam? Output: no or yes.
Fraud detection: is this online transaction fraudulent (for example, a stolen credit card)? Output: no or yes.
Medical diagnosis: is this tumor malignant? Output: no or yes.

In each case, the variable you want to predict has only two possible outcomes. That is binary classification: the word binary means there are only two classes or categories.

Review Questions

1. Is predicting whether an email is spam regression or classification?

Answer

Classification. The output is a category (spam or not spam), not a continuous number.

Labels for Binary Classification

In these problems, class and category mean basically the same thing. By convention, the two classes are often written as:

Convention	Negative class	Positive class
Words	no	yes
Boolean	false	true
Numeric	0	1

In code and formulas, this course usually uses 0 and 1 because that fits cleanly with the algorithms you will implement. In prose, we still say no/yes or benign/malignant when that reads more naturally.

Negative and Positive Class

Another common convention:

Negative class: false or 0 (for example, not spam, not fraudulent, benign tumor)
Positive class: true or 1 (spam present, fraud present, malignant tumor)

Negative and positive do not necessarily mean bad versus good. They convey absence (0, false) versus presence (1, true) of the property you are detecting.

Which class is 0 and which is 1 is sometimes arbitrary. One engineer might call a legitimate email the positive class; another might swap the labels. Either choice can work if you are consistent.

Review Questions

1. In spam detection, why might a non-spam email be called a “negative” example?

Answer

The answer to “is it spam?” is no (0). Negative here means the property you are looking for (spam) is absent, not that the email is bad.

1. Why does this course often use 0 and 1 for class labels in code?

Answer

Numeric 0/1 labels align with the math and Python arrays used in classification algorithms, while still mapping cleanly to no/yes in applications.

Tumor Size: A First Attempt with Linear Regression

How do you build a classification algorithm? Start with a training set for malignant versus benign tumors. Plot tumor size on the horizontal axis and label \(y\) on the vertical axis (0 or 1).

import numpy as np

# Tumor size (cm) vs malignant/benign label
x_train = np.array([0.0, 1, 2, 3, 4, 5])
y_train = np.array([0, 0, 0, 1, 1, 1])

One approach is to apply what you already know: fit a straight line with linear regression so \(f_{w,b}(x) = wx + b\).

Linear regression does not only output 0 and 1. It predicts any real number, including values below 0 or above 1. That is already a hint that something is mismatched.

Review Questions

1. Can \(f_{w,b}(x) = wx + b\) always return only 0 or 1?

Answer

No. A linear function can return any real value. Classification needs discrete class labels, not an unbounded number line.

Threshold at 0.5

One workaround is to pick a threshold, such as 0.5:

If \(f_{w,b}(x) < 0.5\), predict \(y = 0\) (benign)
If \(f_{w,b}(x) \ge 0.5\), predict \(y = 1\) (malignant)

The threshold marks a decision boundary: tumors smaller than the cutoff are classified one way; larger ones the other way.

On this small data set, the threshold can look reasonable. The problem appears when you add one more training example.

Review Questions

1. If \(f_{w,b}(3) = 0.64\), what class do you predict with a 0.5 threshold?

Answer

\(y = 1\) (malignant), because \(0.64 \ge 0.5\).

Why Linear Regression Fails for Classification

Add one more malignant example far to the right (large tumor size). That point should not change how you classify typical tumors. The old vertical cutoff still makes sense.

But the best-fit line shifts. With the same 0.5 threshold, the decision boundary moves right, and a tumor at size 3 can flip from malignant to benign.

x_out = np.append(x_train, 10.0)
y_out = np.append(y_train, 1)
w_bad, b_bad = fit_linear_gd(x_out, y_out)
thr_bad = (0.5 - b_bad) / w_bad
print(f"Without outlier: threshold x ≈ {(0.5 - b_lr) / w_lr:.2f}, f(3) ≈ {w_lr * 3 + b_lr:.2f}")
print(f"With outlier:    threshold x ≈ {thr_bad:.2f}, f(3) ≈ {w_bad * 3 + b_bad:.2f}")

Without outlier: threshold x ≈ 2.37, f(3) ≈ 0.64
With outlier:    threshold x ≈ 3.13, f(3) ≈ 0.48

Large tumors should stay malignant. Linear regression lets a single far-away point drag the line and the cutoff enough to misclassify mid-size tumors. That is why Andrew Ng does not recommend linear regression for classification.

Review Questions

1. After adding a malignant point at very large size, why can a tumor at \(x = 3\) be misclassified?

Answer

The fitted line rotates downward to fit the outlier. The 0.5 threshold crossing moves right, and \(f_{w,b}(3)\) can drop below 0.5 even though the label at \(x = 3\) is still malignant.

Logistic Regression Preview

You will learn more about the decision boundary in the next section. The fix is logistic regression: the model output is always between 0 and 1, so it behaves like a probability of the positive class and avoids the wild extrapolation of a straight line.

One confusing detail: despite the word regression in the name, logistic regression is used for classification (historical naming). It targets binary labels \(y \in \{0, 1\}\).

Review Questions

1. True or false? Logistic regression is mainly used to predict continuous dollar amounts.

Answer

False. It is used for binary classification, even though the name contains “regression.”

Lab: Classification

In this lab, you will contrast regression and classification. You will plot yes/no data, then try fitting a straight line to tumor labels and see where it breaks.

Goals

In this lab, you will:

explore categorical data sets and plotting
try linear regression on classification labels with an interactive plot
see why a straight line is not enough for yes/no problems

Tools

You will use NumPy, Matplotlib, and an interactive plot below (the same tumor-size example from the lecture).

Plotting Classification Data

Examples of classification problems include spam or not spam, fraud or not fraud, and malignant or benign tumors. In binary classification there are only two outcomes, often written as 0/1, no/yes, or false/true.

Classification plots often use symbols: X for the positive class (\(y = 1\)) and O for the negative class (\(y = 0\)).

import numpy as np
import matplotlib.pyplot as plt

plt.style.use("../../deeplearning.mplstyle")

x_lab = np.array([0.0, 1, 2, 3, 4, 5])
y_lab = np.array([0, 0, 0, 1, 1, 1])
X_lab2 = np.array([[0.5, 1.5], [1, 1], [1.5, 0.5], [3, 0.5], [2, 2], [1, 2.5]])
y_lab2 = np.array([0, 0, 0, 1, 1, 1])

One feature: positive examples appear as red X marks at \(y = 1\); negative examples are blue circles at \(y = 0\). In linear regression, \(y\) could be any number, not just 0 or 1.

Two features: there is no \(y\) axis. Positive points are X marks and negative points are O marks in the \((x_0, x_1)\) plane.

Linear Regression on Classification Labels

The interactive plot below uses the tumor-size data from this page. Work through these steps:

Click Run Linear Regression and watch the best-fit line update in stages (this mirrors the animated fit in the original lab).
Notice the line does not match the 0/1 labels very well.
Check Toggle 0.5 threshold to shade benign vs malignant regions. With only the original six points, the threshold can look reasonable.
Click on the plot to add new points. Click in the upper half (\(y > 0.5\)) to add a malignant point; click in the lower half to add a benign point. Try adding a malignant point at large tumor size (near \(x = 10\)).
Run linear regression again. The line shifts, and a tumor at \(x = 3\) can be misclassified.
Click Reset data to restore the original six points.

{
  const dlblue = "#0096ff";
  const dlorange = "#FF9300";
  const dldarkred = "#C00000";

  const initX = [0, 1, 2, 3, 4, 5];
  const initY = [0, 0, 0, 1, 1, 1];

  let ptsX = initX.slice();
  let ptsY = initY.slice();
  let w = 0;
  let b = 0;
  let showThresh = false;
  let lineLabel = "";

  const xMin = -0.5;
  const xMax = 12;
  const yMin = -0.05;
  const yMax = 1.1;
  const plotW = 560;
  const plotH = 280;
  const margin = {top: 28, right: 18, bottom: 42, left: 48};
  const innerW = plotW - margin.left - margin.right;
  const innerH = plotH - margin.top - margin.bottom;

  const xScale = d3.scaleLinear().domain([xMin, xMax]).range([0, innerW]);
  const yScale = d3.scaleLinear().domain([yMin, yMax]).range([innerH, 0]);

  function gradientDescent(x, y, w0, b0, alpha, numIters) {
    let wv = w0, bv = b0;
    const m = x.length;
    for (let t = 0; t < numIters; t++) {
      let dj_dw = 0, dj_db = 0;
      for (let i = 0; i < m; i++) {
        const err = wv * x[i] + bv - y[i];
        dj_dw += err * x[i];
        dj_db += err;
      }
      dj_dw /= m;
      dj_db /= m;
      wv -= alpha * dj_dw;
      bv -= alpha * dj_db;
    }
    return {w: wv, b: bv};
  }

  const container = html`<div class="lr-class-tool" style="max-width:720px;margin:1rem 0;">
    <div style="display:flex;flex-wrap:wrap;gap:0.75rem;align-items:center;margin-bottom:0.75rem;">
      <button id="run-lr" style="padding:0.45rem 0.9rem;background:${dlblue};color:white;border:none;border-radius:4px;cursor:pointer;font-size:0.95rem;">Run Linear Regression</button>
      <label style="display:flex;align-items:center;gap:0.4rem;font-size:0.95rem;cursor:pointer;">
        <input type="checkbox" id="thresh-toggle" />
        Toggle 0.5 threshold (after regression)
      </label>
      <button id="reset-data" style="padding:0.45rem 0.9rem;background:#666;color:white;border:none;border-radius:4px;cursor:pointer;font-size:0.95rem;">Reset data</button>
    </div>
    <svg id="lr-svg" width="${plotW}" height="${plotH}" style="max-width:100%;height:auto;border:1px solid #ddd;border-radius:4px;cursor:crosshair;background:white;"></svg>
    <p style="font-size:0.9rem;color:#555;margin-top:0.5rem;">Click inside the plot to add a data point. Upper half adds malignant (y=1); lower half adds benign (y=0).</p>
    <p id="lr-status" style="font-size:0.9rem;color:#333;margin-top:0.25rem;"></p>
  </div>`;

  const svg = d3.select(container.querySelector("#lr-svg"));
  const g = svg.append("g").attr("transform", `translate(${margin.left},${margin.top})`);
  const gridG = g.append("g");
  const threshG = g.append("g");
  const lineG = g.append("g");
  const ptsG = g.append("g");
  const labelG = g.append("g");

  function drawGrid() {
    gridG.selectAll("*").remove();
    for (let xv = 0; xv <= 10; xv += 2) {
      gridG.append("line").attr("x1", xScale(xv)).attr("x2", xScale(xv)).attr("y1", 0).attr("y2", innerH).attr("stroke", "#eee");
    }
    for (let yv = 0; yv <= 1; yv += 0.25) {
      gridG.append("line").attr("x1", 0).attr("x2", innerW).attr("y1", yScale(yv)).attr("y2", yScale(yv)).attr("stroke", "#eee");
    }
    gridG.append("line").attr("x1", 0).attr("x2", innerW).attr("y1", yScale(0)).attr("y2", yScale(0)).attr("stroke", "#333").attr("stroke-width", 0.8);
    gridG.append("line").attr("x1", xScale(0)).attr("x2", xScale(0)).attr("y1", 0).attr("y2", innerH).attr("stroke", "#333").attr("stroke-width", 0.8);
    gridG.append("text").attr("x", innerW / 2).attr("y", innerH + 32).attr("text-anchor", "middle").attr("font-size", 12).text("Tumor Size");
    gridG.append("text").attr("transform", "rotate(-90)").attr("x", -innerH / 2).attr("y", -34).attr("text-anchor", "middle").attr("font-size", 12).text("y");
    gridG.append("text").attr("x", innerW - 4).attr("y", 14).attr("text-anchor", "end").attr("font-size", 11).attr("fill", "#888").text("[Click to add data points]");
  }

  function drawPoints() {
    ptsG.selectAll("*").remove();
    for (let i = 0; i < ptsX.length; i++) {
      const px = xScale(ptsX[i]);
      const py = yScale(ptsY[i]);
      if (ptsY[i] === 1) {
        ptsG.append("line").attr("x1", px - 5).attr("x2", px + 5).attr("y1", py - 5).attr("y2", py + 5).attr("stroke", dldarkred).attr("stroke-width", 2.5);
        ptsG.append("line").attr("x1", px - 5).attr("x2", px + 5).attr("y1", py + 5).attr("y2", py - 5).attr("stroke", dldarkred).attr("stroke-width", 2.5);
      } else {
        ptsG.append("circle").attr("cx", px).attr("cy", py).attr("r", 7).attr("fill", "none").attr("stroke", dlblue).attr("stroke-width", 2.5);
      }
    }
  }

  function drawLine() {
    lineG.selectAll("*").remove();
    if (!lineLabel) return;
    const x0 = xMin;
    const x1 = xMax;
    const y0 = w * x0 + b;
    const y1 = w * x1 + b;
    lineG.append("line")
      .attr("x1", xScale(x0)).attr("y1", yScale(y0))
      .attr("x2", xScale(x1)).attr("y2", yScale(y1))
      .attr("stroke", dlblue).attr("stroke-width", 2.5);
    labelG.selectAll("*").remove();
    labelG.append("text")
      .attr("x", innerW - 8).attr("y", innerH - 8)
      .attr("text-anchor", "end").attr("font-size", 11).attr("fill", dlblue)
      .text(lineLabel);
  }

  function drawThreshold() {
    threshG.selectAll("*").remove();
    if (!showThresh || Math.abs(w) < 1e-8) return;
    const xCut = (0.5 - b) / w;
    threshG.append("rect").attr("x", xScale(xMin)).attr("y", 0).attr("width", xScale(xCut) - xScale(xMin)).attr("height", innerH).attr("fill", dlblue).attr("opacity", 0.12);
    threshG.append("rect").attr("x", xScale(xCut)).attr("y", 0).attr("width", xScale(xMax) - xScale(xCut)).attr("height", innerH).attr("fill", dldarkred).attr("opacity", 0.12);
    threshG.append("line").attr("x1", xScale(xCut)).attr("x2", xScale(xCut)).attr("y1", 0).attr("y2", innerH).attr("stroke", "#888").attr("stroke-dasharray", "4,3");
    threshG.append("text").attr("x", xScale(xCut) + 8).attr("y", yScale(0.55)).attr("font-size", 11).attr("fill", dldarkred).text("Malignant");
    threshG.append("text").attr("x", xScale(xCut) - 8).attr("y", yScale(0.55)).attr("text-anchor", "end").attr("font-size", 11).attr("fill", dlblue).text("Benign");
  }

  function updateStatus(msg) {
    container.querySelector("#lr-status").textContent = msg;
  }

  function redraw() {
    drawPoints();
    drawLine();
    drawThreshold();
    if (lineLabel && Math.abs(w) > 1e-8) {
      const f3 = w * 3 + b;
      updateStatus(`Model: ${lineLabel}.  f(3) ≈ ${f3.toFixed(2)}.  Threshold cutoff x ≈ ${((0.5 - b) / w).toFixed(2)}.`);
    } else if (!lineLabel) {
      updateStatus("Click Run Linear Regression to fit a line to the current points.");
    }
  }

  drawGrid();
  redraw();

  container.querySelector("#run-lr").addEventListener("click", async () => {
    const btn = container.querySelector("#run-lr");
    btn.disabled = true;
    btn.textContent = "Running...";
    const steps = [1, 1, 1, 1, 1, 2, 4, 8, 16, 32, 64, 128, 256];
    for (const it of steps) {
      ({w, b} = gradientDescent(ptsX, ptsY, w, b, 0.01, it));
      lineLabel = `y = ${w.toFixed(2)}x + (${b.toFixed(2)})`;
      redraw();
      await new Promise(r => setTimeout(r, 280));
    }
    btn.disabled = false;
    btn.textContent = "Run Linear Regression";
  });

  container.querySelector("#thresh-toggle").addEventListener("change", (ev) => {
    showThresh = ev.target.checked;
    redraw();
  });

  container.querySelector("#reset-data").addEventListener("click", () => {
    ptsX = initX.slice();
    ptsY = initY.slice();
    w = 0;
    b = 0;
    lineLabel = "";
    showThresh = false;
    container.querySelector("#thresh-toggle").checked = false;
    redraw();
  });

  svg.on("click", (event) => {
    const [mx, my] = d3.pointer(event, g.node());
    if (mx < 0 || mx > innerW || my < 0 || my > innerH) return;
    const xVal = xScale.invert(mx);
    const yVal = yScale.invert(my) > 0.5 ? 1 : 0;
    ptsX = [...ptsX, xVal];
    ptsY = [...ptsY, yVal];
    redraw();
  });

  return container;
}

Lab Summary

In this lab you:

plotted one-variable and two-variable classification data sets
ran linear regression on tumor labels and toggled a 0.5 threshold
added an outlier and saw the decision boundary shift so mid-size tumors can be misclassified
saw that linear regression alone is not enough for classification problems

What Comes Next

The next page introduces logistic regression for classification, where the output stays between 0 and 1 and avoids the problems shown above. (Despite the name, logistic regression is used for classification, not for predicting arbitrary numbers.)