The Problem of Overfitting

You have now seen linear regression and logistic regression. Both work well on many tasks. Sometimes, though, a model can overfit the training data and…

You have now seen linear regression and logistic regression. Both work well on many tasks. Sometimes, though, a model can overfit the training data and perform poorly on new examples. This page explains overfitting and its close opposite, underfitting.

The next page will introduce regularization, a widely used technique for reducing overfitting. If you have taken the statistics course, you may already have seen overfitting and regularization in the context of polynomial models there.

Review Questions

1. What is the opposite problem to overfitting?

TipAnswer

Underfitting, where the model is too simple to capture the pattern in the training data.

Underfitting and Overfitting

A model can fail in two different ways.

• Underfitting means the model is too simple. It does not even fit the training set well.
• Overfitting means the model fits the training set almost too well, often by being overly complex. It may fail on new data.

In machine learning, people often pair these terms with bias and variance (different usage from “variance” in probability).

Problem	Common paired term
Underfitting	High bias
Overfitting	High variance

Review Questions

1. If a model has high variance, is it more likely to be underfitting or overfitting?

TipAnswer

Overfitting. High variance means small changes in the training data can lead to large changes in the fitted model.

Housing Prices Example

To make these ideas concrete, return to predicting house prices with linear regression. The input \(x\) is house size. The target \(y\) is price.

Suppose you have only five training examples. The scatter below shows size on the horizontal axis and price on the vertical axis. As size grows, prices rise at first and then flatten out.

Three models below use the same five points.

1. Linear (one feature \(x\))
2. Quadratic (features \(x\) and \(x^2\)), as in polynomial regression
3. Fourth-order polynomial (features \(x, x^2, x^3, x^4\))

Underfitting (High Bias)

The linear model on the left is too rigid. It assumes price grows steadily with size, but the data flatten. The straight line misses the pattern in the training set.

This is underfitting, also called high bias. The model has a strong preconception (prices must be a straight-line function of size) even though the data suggest otherwise.

You may have read news stories about machine learning systems showing unfair bias against certain groups. Checking models for that kind of bias is essential. Here, high bias means a technical underfitting problem, not fairness. The word bias has both meanings in machine learning.

Generalization

Real estate agents care about prices on new houses, not only the five houses in the training set. You want the model to do well on examples it has never seen. That ability is called generalization.

The quadratic model in the middle fits the training points reasonably well without becoming too wiggly. It is a plausible model for new houses of similar size.

Overfitting (High Variance)

The fourth-order curve on the right passes through all five training points almost perfectly. Training error can be driven very close to zero.

But the curve is wiggly. For some sizes, it predicts a larger house is cheaper than a smaller one, which is not reasonable. The model memorized the training points instead of learning a smooth trend.

This is overfitting, also called high variance. If the training set changed slightly (one house priced a little higher or lower), the fitted curve could change a lot. That sensitivity is why people say the model has high variance.

The statistics course shows the same pattern with higher-degree polynomials. Complex models can achieve very low training loss yet fail on new data. That is the overfitting problem in a nutshell.

Review Questions

1. Why can a fourth-order polynomial have training MSE near zero on only five points?

TipAnswer

With enough parameters, the model can bend enough to pass through every training point. That does not mean it will predict well on new houses.

---

1. Which model in the plot is most likely to generalize well to new houses?

TipAnswer

The quadratic model in the middle. It captures the flattening trend without being too wiggly.

Goldilocks Analogy

Think of the children’s story Goldilocks and the Three Bears.

• One bowl of porridge is too cold (underfit, high bias).
• Another is too hot (overfit, high variance).
• One is just right in the middle.

Your goal in machine learning is similar. You want a model that is neither too simple nor too complex.

• Too few features (like the linear model) can underfit.
• Too many features (like the fourth-order polynomial) can overfit.
• A middle complexity (like quadratic features \(x\) and \(x^2\)) is often a good starting point.

Review Questions

1. In the housing example, what feature setup matches the “just right” bowl of porridge?

TipAnswer

Using quadratic features (\(x\) and \(x^2\)), which fits the flattening curve without being too wiggly.

Classification Example

Overfitting is not limited to regression. It appears in classification too.

Suppose you classify tumors as malignant (\(y = 1\), red X) or benign (\(y = 0\), blue circle) using two features.

• \(x_1\) might be tumor size
• \(x_2\) might be patient age

Three logistic regression models use different feature sets.

1. Linear features only (\(x_1\), \(x_2\)) give a straight decision boundary.
2. Quadratic features (including \(x_1^2\), \(x_2^2\), \(x_1 x_2\)) can bend the boundary into an ellipse-like shape.
3. A high-degree polynomial feature map lets the boundary twist to fit every training point.

The linear boundary on the left is not terrible, but it does not separate the clusters well. This is underfitting (high bias).

The quadratic boundary in the middle misclassifies a few training points but follows a smooth shape that should generalize to new patients.

The high-degree boundary on the right can contort itself to classify every training example correctly. That looks like strong training performance, but the boundary is overly complex for tumor size and age alone. This is overfitting (high variance).

Recall from logistic regression that with only original features \(x_1, x_2\), the decision boundary is linear. Polynomial features let it bend, which helps until you add so many terms that the model starts to memorize noise.

Review Questions

1. True or false? With only features \(x_1\) and \(x_2\), logistic regression always has a linear decision boundary.

TipAnswer

True for the original features. To get curved boundaries you need nonlinear feature terms (such as \(x_1^2\) or \(x_1 x_2\)).

---

1. Why might perfect accuracy on the training set still be a bad sign?

TipAnswer

The model may be overfitting, fitting noise or quirks of the training set instead of a pattern that generalizes to new patients.

---

1. You fit logistic regression with polynomial features to a dataset, and the decision boundary looks like the high-degree (overfit) panel in the classification plots above. What would you conclude?

1. The model has high variance (overfit). Thus, adding data is, by itself, unlikely to help much.

2. The model has high variance (overfit). Thus, adding data is likely to help.

3. The model has high bias (underfit). Thus, adding data is, by itself, unlikely to help much.

4. The model has high bias (underfit). Thus, adding data is likely to help.

TipAnswer

b. The boundary is overly complex and fits the training set too closely, which is high variance (overfit). When overfitting is the main problem, collecting more training examples often helps the model learn a smoother pattern.

Goal

The goal is a model that is just right.

• Not underfitting (high bias)
• Not overfitting (high variance)

In practice you judge this by training error and performance on data the model did not see during training (a validation or test set).

Later in this specialization you will learn tools for debugging learning algorithms and recognizing when overfitting or underfitting is happening. For now, suppose you fit a model and it has high variance (it overfits). What can you do?

Addressing Overfitting

Return to the degree-4 housing curve from earlier. It bends sharply to hit every training point. One way to tame that wiggly curve is to give the learning algorithm more examples.

More Training Data

The number one tool against overfitting, when you can use it, is to collect more training data.

If you can record more house sales (more sizes and prices), the algorithm can still fit a flexible model (high-degree polynomial or many features), but with enough examples it will tend to learn a smoother trend instead of memorizing noise.

More data is not always possible. In a small neighborhood, only so many houses may have sold recently. When additional data is available, though, it often works very well.

Fewer Features

A second option is to use fewer features.

In the housing example, the overfit model used \(x\), \(x^2\), \(x^3\), and \(x^4\). You could drop the highest-degree terms and keep only \(x\) and \(x^2\), as in the polynomial regression section.

Now imagine a richer data set with many input features per house, such as size, number of bedrooms, number of floors, age, average neighborhood income, and distance to the nearest coffee shop. If you have 100 features but only a modest number of training examples, the model may overfit.

Instead of using all 100 features, you might keep a smaller subset that seems most relevant, such as size, bedrooms, and age. Picking the most useful features is sometimes called feature selection. You can use domain intuition, or later you will see algorithms that help choose features automatically.

Feature selection has a trade-off. Dropping features throws away information. Perhaps all 100 inputs are genuinely useful for price prediction, and you do not want to discard them lightly.

Regularization (Preview)

The third option is regularization, which the next page develops in full. If you have seen \(L_2\) regularization in the statistics course, you already have a head start.

Look again at an overfit polynomial model with features \(x\), \(x^2\), \(x^3\), \(x^4\), and so on. The fitted weights \(w_1, w_2, \ldots\) are often large. Dropping the \(x^4\) term means setting \(w_4 = 0\), which is the same as eliminating that feature entirely.

Regularization takes a gentler approach. It encourages the algorithm to shrink the weights without forcing them to be exactly zero. Even with high-degree terms present, smaller \(w_j\) values produce a less wiggly curve that may generalize better.

By convention, regularization usually penalizes \(w_1, w_2, \ldots, w_n\) but not the bias \(b\). In practice, whether you regularize \(b\) as well makes little difference.

Three Options Recap

When a model overfits, you can try the following.

1. Collect more training data (best when feasible).
2. Use fewer features (feature selection).
3. Apply regularization to keep all features but limit how large their weights can grow.

Regularization is widely used in practice, including for neural networks later in this specialization. The next page shows how to add it to linear regression and logistic regression.

Review Questions

1. If you can obtain more labeled training examples, which overfitting remedy is usually the first choice?

TipAnswer

Collect more training data. More examples often let a flexible model learn a smoother pattern instead of fitting noise.

---

1. How is setting \(w_4 = 0\) in a polynomial model related to feature selection?

TipAnswer

It removes the \(x^4\) term from the model, which is equivalent to not using that feature.

---

1. What technique does the next page introduce in detail to address overfitting?

TipAnswer

Regularization, which penalizes large weights so the model stays flexible without overfitting as severely.

---

1. Our goal when creating a model is to be able to use the model to predict outcomes correctly for new examples. A model which does this is said to generalize well.

When a model fits the training data well but does not work well with new examples that are not in the training set, this is an example of:

1. Underfitting (high bias)

2. None of the above

3. Overfitting (high variance)

4. A model that generalizes well (neither high variance nor high bias)

TipAnswer

c. Overfitting (high variance). The model fits the training set closely but fails on new data. That is the opposite of generalizing well.

---

1. Which of the following is not one of the three main options for reducing overfitting discussed on this page?

1. Collect more training data

2. Use fewer features

3. Apply regularization

4. Increase the learning rate \(\alpha\) to converge faster

TipAnswer

d. The three options are more data, fewer features, and regularization. Changing the learning rate affects optimization speed, not overfitting directly.

---

1. Which of the following can address overfitting? (Select all that apply.)

1. Collect more training data

2. Apply regularization

3. Remove a random set of training examples

4. Select a subset of the more relevant features

TipAnswer

a, b, and d. More training data, regularization, and feature selection can all reduce overfitting. Removing a random subset of training examples does not help. You want more data or better-chosen features, not arbitrary deletions.

Lab: Overfitting

Polynomial models can be powerful, but they are also easy to misuse. In this lab you will fit curves and decision boundaries at different degrees, watch underfitting and overfitting appear, and then try regularization (\(\lambda\)) as a remedy. The interactive plot runs in your browser (Observable JavaScript and D3), so you can experiment without writing a full training loop first.

Goals

In this lab, you will:

• fit regression and classification data at polynomial degrees 1 through 6
• see underfitting at low degree and overfitting at high degree
• add points and watch how outliers or extra examples change the fit
• turn regularization on and off (\(\lambda\)) and compare fits at the same degree

Tools

You will use the interactive plot below. It uses normalized polynomial features and fast solvers (normal equations for regression, gradient descent for classification), similar to the reference labs.

Explore Overfitting Interactively

The plot below lets you:

• switch between Regression and Categorical examples
• select the model degree (1 through 6)
• click Fit data to train a polynomial model on the current points
• click on the plot to add data (hold Shift while clicking in classification mode to add a blue \(y = 0\) point)
• click Reset to restore the original data set

Here are some things you should try:

• Fit with degree = 1 and note underfitting.
• Fit with degree = 6 and note overfitting.
• Tune the degree to find a fit that looks just right.
• Add data: extreme points can increase overfitting (if they are outliers); typical points can reduce it.
• Switch between Regression and Categorical to compare both settings.
• Regularization experiment (classification). Set degree = 6, \(\lambda = 0\), click Fit data, and note the twisted boundary. Then set \(\lambda = 1\), fit again, and compare.
• Regularization experiment (regression). Repeat the same \(\lambda = 0\) vs \(\lambda = 1\) comparison in regression mode.

Notes:

• The dashed orange ideal curve is the noise-free generator model.
• Fit uses normalized polynomial features. Regression uses regularized normal equations; classification uses regularized gradient descent.
• \(\lambda = 0\) means no penalty. Larger \(\lambda\) shrinks weights and usually produces a smoother curve or boundary.

{
  // deeplearning.mplstyle palette and typography
  const dlblue = "#0096FF";
  const dlorange = "#FF9300";
  const dldarkred = "#C00000";
  const dlgreen = "#2ca02c";
  const mplFont = "Verdana, DejaVu Sans, sans-serif";
  const mplLabelSize = 14;
  const mplTitleSize = 11;
  const mplLegendSize = 12;
  const mplAxisColor = "#f0f0f0";

  function munge(base, exp) {
    if (exp === 0) return "";
    if (exp === 1) return base;
    return base + "^{" + exp + "}";
  }

  function mapOneFeature(xArr, degree) {
    const cols = [];
    const terms = [];
    let k = 0;
    for (let i = 1; i <= degree; i++) {
      cols.push(xArr.map(x => Math.pow(x, i)));
      terms.push("w_{" + k + "}" + munge("x", i));
      k += 1;
    }
    terms.push("b");
    const n = xArr.length;
    const X = Array.from({length: n}, (_, row) => cols.map(col => col[row]));
    return {X, equation: terms.join(" + ")};
  }

  function mapFeature(x0Arr, x1Arr, degree) {
    const cols = [];
    const terms = [];
    let k = 0;
    for (let i = 1; i <= degree; i++) {
      for (let j = 0; j <= i; j++) {
        const p0 = i - j;
        const p1 = j;
        cols.push(x0Arr.map((x0, idx) => Math.pow(x0, p0) * Math.pow(x1Arr[idx], p1)));
        terms.push("w_{" + k + "}" + munge("x_0", p0) + munge("x_1", p1));
        k += 1;
      }
    }
    terms.push("b");
    const n = x0Arr.length;
    const X = Array.from({length: n}, (_, row) => cols.map(col => col[row]));
    return {X, equation: terms.join(" + ")};
  }

  function zscoreNormalize(X) {
    const n = X.length;
    if (n === 0) return {X: [], mu: [], sigma: []};
    const d = X[0].length;
    const mu = new Array(d).fill(0);
    const sigma = new Array(d).fill(0);
    for (let j = 0; j < d; j++) {
      let s = 0;
      for (let i = 0; i < n; i++) s += X[i][j];
      mu[j] = s / n;
    }
    for (let j = 0; j < d; j++) {
      let s2 = 0;
      for (let i = 0; i < n; i++) s2 += (X[i][j] - mu[j]) ** 2;
      sigma[j] = Math.sqrt(s2 / n) || 1;
    }
    const Xs = X.map(row => row.map((v, j) => (v - mu[j]) / sigma[j]));
    return {X: Xs, mu, sigma};
  }

  function scaleRows(Xraw, mu, sigma) {
    return Xraw.map(row => row.map((v, j) => (v - mu[j]) / sigma[j]));
  }

  function solveLinear(X, y, lambda_) {
    const m = y.length;
    const n = X[0].length;
    const p = n + 1;
    const ata = Array.from({length: p}, () => new Array(p).fill(0));
    const aty = new Array(p).fill(0);
    for (let i = 0; i < m; i++) {
      const row = [1, ...X[i]];
      for (let a = 0; a < p; a++) {
        aty[a] += row[a] * y[i];
        for (let b = 0; b < p; b++) ata[a][b] += row[a] * row[b];
      }
    }
    for (let j = 1; j < p; j++) ata[j][j] += lambda_ / m;
    for (let col = 0; col < p; col++) {
      let pivot = ata[col][col];
      if (Math.abs(pivot) < 1e-12) return {w: new Array(n).fill(0), b: 0};
      for (let j = col; j < p; j++) ata[col][j] /= pivot;
      aty[col] /= pivot;
      for (let row = 0; row < p; row++) {
        if (row === col) continue;
        const factor = ata[row][col];
        for (let j = col; j < p; j++) ata[row][j] -= factor * ata[col][j];
        aty[row] -= factor * aty[col];
      }
    }
    return {w: aty.slice(1), b: aty[0]};
  }

  function sigmoid(z) {
    if (z > 500) return 1;
    if (z < -500) return 0;
    return 1 / (1 + Math.exp(-z));
  }

  function predictLogistic(X, w, b) {
    return X.map(row => {
      let z = b;
      for (let j = 0; j < w.length; j++) z += w[j] * row[j];
      return sigmoid(z);
    });
  }

  function logisticGD(X, y, alpha, iters, lambda_) {
    const m = y.length;
    const n = X[0].length;
    let w = new Array(n).fill(0);
    let b = 0;
    for (let t = 0; t < iters; t++) {
      const dw = new Array(n).fill(0);
      let db = 0;
      for (let i = 0; i < m; i++) {
        let z = b;
        for (let j = 0; j < n; j++) z += w[j] * X[i][j];
        const f = sigmoid(z);
        const err = f - y[i];
        for (let j = 0; j < n; j++) dw[j] += err * X[i][j];
        db += err;
      }
      for (let j = 0; j < n; j++) {
        dw[j] = dw[j] / m + (lambda_ / m) * w[j];
        w[j] -= alpha * dw[j];
      }
      b -= alpha * db / m;
    }
    return {w, b};
  }

  const regInitX = x_reg_init.slice();
  const regInitY = y_reg_init.slice();
  const clfInitX0 = x_clf_init_0.slice();
  const clfInitX1 = x_clf_init_1.slice();
  const clfInitY = y_clf_init.slice();

  let mode = "categorical";
  let degree = 1;
  let lambda_ = 0;
  const lambdaChoices = [0, 0.1, 1, 10];
  let ptsX = [];
  let ptsY = [];
  let ptsX1 = [];
  let fitLine = null;
  let contourPaths = [];
  let wFit = null;
  let bFit = 0;
  let muFit = null;
  let sigmaFit = null;
  let xLim = [0, 29];
  let yLim = [0, 900];

  const plotW = 620;
  const plotH = 380;
  const margin = {top: 36, right: 22, bottom: 48, left: 54};
  const innerW = plotW - margin.left - margin.right;
  const innerH = plotH - margin.top - margin.bottom;

  function loadCategorical() {
    ptsX = clfInitX0.slice();
    ptsX1 = clfInitX1.slice();
    ptsY = clfInitY.slice();
    fitLine = null;
    contourPaths = [];
    wFit = null;
    xLim = [-1, 1];
    yLim = [-1, 1];
  }

  function loadRegression() {
    ptsX = regInitX.slice();
    ptsY = regInitY.slice();
    ptsX1 = [];
    fitLine = null;
    contourPaths = [];
    wFit = null;
    xLim = [0, 29];
    yLim = [Math.min(...regInitY) - 50, Math.max(...regInitY) + 50];
  }

  loadCategorical();

  function equationText() {
    if (mode === "categorical") {
      const {equation} = mapFeature(ptsX.length ? ptsX : [0], ptsX1.length ? ptsX1 : [0], degree);
      return "f_{wb} = \\mathrm{sigmoid}(" + equation + ")";
    }
    const {equation} = mapOneFeature(ptsX.length ? ptsX : [0], degree);
    return "f_{wb} = (" + equation + ")";
  }

  function makeScales() {
    const xScale = d3.scaleLinear().domain(xLim).range([0, innerW]);
    const yScale = d3.scaleLinear().domain(yLim).range([innerH, 0]);
    return {xScale, yScale};
  }

  const container = html`<div class="overfit-tool" style="max-width:760px;margin:1rem 0;">
    <div style="display:flex;flex-wrap:wrap;gap:0.5rem;align-items:center;margin-bottom:0.6rem;">
      <span style="font-size:0.9rem;font-weight:600;margin-right:0.25rem;">Type</span>
      <button class="mode-btn" data-mode="regression" style="padding:0.35rem 0.75rem;border:1px solid #ccc;border-radius:4px;background:#f5f5f5;cursor:pointer;">Regression</button>
      <button class="mode-btn" data-mode="categorical" style="padding:0.35rem 0.75rem;border:1px solid #ccc;border-radius:4px;background:#f5f5f5;cursor:pointer;">Categorical</button>
    </div>
    <div style="display:flex;flex-wrap:wrap;gap:0.35rem;align-items:center;margin-bottom:0.6rem;">
      <span style="font-size:0.9rem;font-weight:600;margin-right:0.25rem;">Degree</span>
      ${[1,2,3,4,5,6].map(d => html`<button class="deg-btn" data-deg="${d}" style="padding:0.3rem 0.55rem;min-width:2rem;border:1px solid #ccc;border-radius:4px;background:#f5f5f5;cursor:pointer;">${d}</button>`)}
      <span style="font-size:0.9rem;font-weight:600;margin-left:0.35rem;margin-right:0.25rem;">&#955;</span>
      ${lambdaChoices.map(l => html`<button class="lam-btn" data-lam="${l}" style="padding:0.3rem 0.55rem;min-width:2.2rem;border:1px solid #ccc;border-radius:4px;background:#f5f5f5;cursor:pointer;">${l}</button>`)}
      <button id="fit-btn" style="margin-left:0.5rem;padding:0.4rem 0.9rem;background:${dlblue};color:white;border:none;border-radius:4px;cursor:pointer;">Fit data</button>
      <button id="reset-btn" style="padding:0.4rem 0.9rem;background:#666;color:white;border:none;border-radius:4px;cursor:pointer;">Reset</button>
    </div>
    <svg id="of-svg" width="${plotW}" height="${plotH}" style="max-width:100%;height:auto;border:1px solid #ddd;border-radius:4px;cursor:crosshair;background:white;"></svg>
    <p id="of-hint" style="font-size:0.9rem;color:#555;margin-top:0.5rem;"></p>
    <div id="of-eq" style="margin-top:0.35rem;overflow-x:auto;"></div>
    <p id="of-status" style="font-size:0.9rem;color:#333;margin-top:0.25rem;"></p>
  </div>`;

  const svg = d3.select(container.querySelector("#of-svg"));
  const g = svg.append("g").attr("transform", `translate(${margin.left},${margin.top})`);
  const gridG = g.append("g");
  const idealG = g.append("g");
  const contourG = g.append("g");
  const lineG = g.append("g");
  const ptsG = g.append("g");
  const legendG = g.append("g");
  const titleG = g.append("g");

  function updateHint() {
    const hint = container.querySelector("#of-hint");
    hint.textContent = mode === "categorical"
      ? "Click on the plot to add data. Hold Shift while clicking to add a blue (y = 0) point."
      : "Click on the plot to add a data point.";
    const eqDiv = container.querySelector("#of-eq");
    eqDiv.replaceChildren(tex`${equationText()}`);
  }

  function styleButtons() {
    container.querySelectorAll(".mode-btn").forEach(btn => {
      const active = btn.dataset.mode === mode;
      btn.style.background = active ? dlblue : "#f5f5f5";
      btn.style.color = active ? "white" : "#333";
      btn.style.borderColor = active ? dlblue : "#ccc";
    });
    container.querySelectorAll(".deg-btn").forEach(btn => {
      const active = +btn.dataset.deg === degree;
      btn.style.background = active ? dlblue : "#f5f5f5";
      btn.style.color = active ? "white" : "#333";
      btn.style.borderColor = active ? dlblue : "#ccc";
    });
    container.querySelectorAll(".lam-btn").forEach(btn => {
      const active = +btn.dataset.lam === lambda_;
      btn.style.background = active ? dlblue : "#f5f5f5";
      btn.style.color = active ? "white" : "#333";
      btn.style.borderColor = active ? dlblue : "#ccc";
    });
  }

  function drawGrid(xScale, yScale) {
    gridG.selectAll("*").remove();
    titleG.selectAll("*").remove();
    gridG.append("rect")
      .attr("x", 0).attr("y", 0)
      .attr("width", innerW).attr("height", innerH)
      .attr("fill", "#ffffff");
    gridG.append("line").attr("x1", 0).attr("x2", innerW).attr("y1", innerH).attr("y2", innerH)
      .attr("stroke", mplAxisColor).attr("stroke-width", 2);
    gridG.append("line").attr("x1", 0).attr("x2", 0).attr("y1", 0).attr("y2", innerH)
      .attr("stroke", mplAxisColor).attr("stroke-width", 2);
    const xLabel = mode === "categorical" ? "x₀" : "x";
    const yLabel = mode === "categorical" ? "x₁" : "y";
    gridG.append("text").attr("x", innerW / 2).attr("y", innerH + 36)
      .attr("text-anchor", "middle").attr("font-size", mplLabelSize)
      .attr("font-family", mplFont).text(xLabel);
    gridG.append("text").attr("transform", "rotate(-90)")
      .attr("x", -innerH / 2).attr("y", -38)
      .attr("text-anchor", "middle").attr("font-size", mplLabelSize)
      .attr("font-family", mplFont).text(yLabel);
    const title = mode === "categorical"
      ? "Overfitting Example: Categorical data set with noise"
      : "Overfitting Example: Regression Data Set (quadratic with noise)";
    titleG.append("text").attr("x", innerW / 2).attr("y", -14)
      .attr("text-anchor", "middle").attr("font-size", mplTitleSize)
      .attr("font-family", mplFont).attr("fill", "gray").text(title);
  }

  function drawIdeal(xScale, yScale) {
    idealG.selectAll("*").remove();
    let idealX, idealY;
    if (mode === "categorical") {
      idealX = ptsX.slice().sort((a, b) => a - b);
      idealY = idealX.map(x => x * x);
    } else {
      idealX = d3.range(xLim[0], xLim[1] + 0.1, 0.5);
      idealY = idealX.map(x => x * x);
    }
    const line = d3.line()
      .x((d, i) => xScale(idealX[i]))
      .y((d, i) => yScale(idealY[i]));
    idealG.append("path")
      .datum(idealY)
      .attr("fill", "none")
      .attr("stroke", dlorange)
      .attr("stroke-width", 1)
      .attr("stroke-dasharray", "6,4")
      .attr("d", line);
    const labelIdx = Math.floor(idealX.length * 0.68);
    idealG.append("text")
      .attr("x", xScale(idealX[labelIdx]))
      .attr("y", yScale(idealY[labelIdx]) - 8)
      .attr("font-size", mplLegendSize)
      .attr("font-family", mplFont)
      .attr("fill", dlorange)
      .attr("stroke", "#ffffff")
      .attr("stroke-width", 3)
      .attr("paint-order", "stroke")
      .text("ideal");
  }

  function drawLegend() {
    legendG.selectAll("*").remove();
    const items = [{label: "ideal", kind: "dashed", color: dlorange}];
    if (mode === "regression") {
      items.unshift({label: "y", kind: "circle", color: dlblue});
      if (fitLine) items.push({label: "y_fit", kind: "solid", color: dlblue});
    } else if (contourPaths.length) {
      items.push({label: "boundary", kind: "solid", color: dlgreen});
    }
    const pad = 8;
    const symW = 24;
    const rowH = 18;
    const textW = Math.max(...items.map(d => d.label.length * 7.2));
    const boxW = pad * 2 + symW + textW;
    const boxH = pad * 2 + items.length * rowH;
    const x0 = innerW - boxW - 4;
    const y0 = innerH - boxH - 4;
    legendG.append("rect")
      .attr("x", x0).attr("y", y0)
      .attr("width", boxW).attr("height", boxH)
      .attr("fill", "#ffffff").attr("fill-opacity", 0.92)
      .attr("stroke", "#cccccc").attr("rx", 4);
    items.forEach((item, i) => {
      const cy = y0 + pad + i * rowH + rowH / 2;
      const lx = x0 + pad;
      if (item.kind === "dashed") {
        legendG.append("line")
          .attr("x1", lx).attr("x2", lx + symW).attr("y1", cy).attr("y2", cy)
          .attr("stroke", item.color).attr("stroke-width", 1)
          .attr("stroke-dasharray", "6,4");
      } else if (item.kind === "solid") {
        legendG.append("line")
          .attr("x1", lx).attr("x2", lx + symW).attr("y1", cy).attr("y2", cy)
          .attr("stroke", item.color).attr("stroke-width", 3);
      } else if (item.kind === "circle") {
        legendG.append("circle")
          .attr("cx", lx + symW / 2).attr("cy", cy).attr("r", 4)
          .attr("fill", "none").attr("stroke", item.color).attr("stroke-width", 2);
      }
      legendG.append("text")
        .attr("x", lx + symW + 6).attr("y", cy + 4)
        .attr("font-size", mplLegendSize)
        .attr("font-family", mplFont)
        .attr("fill", "#000000")
        .text(item.label);
    });
  }

  function drawPoints(xScale, yScale) {
    ptsG.selectAll("*").remove();
    for (let i = 0; i < ptsX.length; i++) {
      const px = xScale(ptsX[i]);
      const py = yScale(mode === "categorical" ? ptsX1[i] : ptsY[i]);
      if (mode === "categorical") {
        if (ptsY[i] === 1) {
          ptsG.append("line").attr("x1", px - 4).attr("x2", px + 4).attr("y1", py - 4).attr("y2", py + 4).attr("stroke", dldarkred).attr("stroke-width", 2);
          ptsG.append("line").attr("x1", px - 4).attr("x2", px + 4).attr("y1", py + 4).attr("y2", py - 4).attr("stroke", dldarkred).attr("stroke-width", 2);
        } else {
          ptsG.append("circle").attr("cx", px).attr("cy", py).attr("r", 5).attr("fill", "none").attr("stroke", dlblue).attr("stroke-width", 2);
        }
      } else {
        ptsG.append("circle").attr("cx", px).attr("cy", py).attr("r", 4).attr("fill", "none").attr("stroke", dlblue).attr("stroke-width", 2);
      }
    }
  }

  function drawFit(xScale, yScale) {
    lineG.selectAll("*").remove();
    contourG.selectAll("*").remove();
    if (!fitLine && !contourPaths.length) return;
    if (mode === "regression" && fitLine) {
      const line = d3.line()
        .x(d => xScale(d.x))
        .y(d => yScale(d.y));
      lineG.append("path")
        .datum(fitLine)
        .attr("fill", "none")
        .attr("stroke", dlblue)
        .attr("stroke-width", 3)
        .attr("d", line);
      const mid = fitLine[Math.floor(fitLine.length * 0.55)];
      lineG.append("text")
        .attr("x", xScale(mid.x))
        .attr("y", yScale(mid.y) - 8)
        .attr("font-size", mplLegendSize)
        .attr("font-family", mplFont)
        .attr("fill", dlblue)
        .attr("stroke", "#ffffff")
        .attr("stroke-width", 3)
        .attr("paint-order", "stroke")
        .text("y_fit");
    }
    if (mode === "categorical" && contourPaths.length) {
      contourPaths.forEach(seg => {
        contourG.append("line")
          .attr("x1", seg[0][0]).attr("y1", seg[0][1])
          .attr("x2", seg[1][0]).attr("y2", seg[1][1])
          .attr("stroke", dlgreen)
          .attr("stroke-width", 2);
      });
    }
  }

  function redraw() {
    const {xScale, yScale} = makeScales();
    drawGrid(xScale, yScale);
    drawIdeal(xScale, yScale);
    drawFit(xScale, yScale);
    drawPoints(xScale, yScale);
    drawLegend();
    styleButtons();
    updateHint();
    const status = container.querySelector("#of-status");
    status.textContent = wFit
      ? `Fitted degree-${degree} model with lambda = ${lambda_} on ${ptsX.length} points.`
      : "Select degree and lambda, then click Fit data to train on the current points.";
  }

  function fitRegression() {
    const {X: Xraw} = mapOneFeature(ptsX, degree);
    const {X, mu, sigma} = zscoreNormalize(Xraw);
    const {w, b} = solveLinear(X, ptsY, lambda_);
    wFit = w;
    bFit = b;
    muFit = mu;
    sigmaFit = sigma;
    const xs = d3.range(xLim[0], xLim[1] + 0.01, (xLim[1] - xLim[0]) / 80);
    const {X: Xm} = mapOneFeature(xs, degree);
    const Xms = scaleRows(Xm, mu, sigma);
    fitLine = xs.map((x, i) => {
      let pred = b;
      for (let j = 0; j < w.length; j++) pred += w[j] * Xms[i][j];
      return {x, y: pred};
    });
  }

  function fitCategorical() {
    const {X: Xraw} = mapFeature(ptsX, ptsX1, degree);
    const {X, mu, sigma} = zscoreNormalize(Xraw);
    const {w, b} = logisticGD(X, ptsY, 0.3, 3000, lambda_);
    wFit = w;
    bFit = b;
    muFit = mu;
    sigmaFit = sigma;
    // Build decision boundary using marching squares (no d3-contour needed)
    const nx = 100, ny = 100;
    const dx = (xLim[1] - xLim[0]) / nx;
    const dy = (yLim[1] - yLim[0]) / ny;
    // Compute grid of predictions
    const grid = [];
    for (let iy = 0; iy <= ny; iy++) {
      const row = [];
      const x1v = yLim[0] + iy * dy;
      for (let ix = 0; ix <= nx; ix++) {
        const x0v = xLim[0] + ix * dx;
        const {X: Xm} = mapFeature([x0v], [x1v], degree);
        const Xms = scaleRows(Xm, mu, sigma);
        const p = predictLogistic(Xms, w, b)[0];
        row.push(p);
      }
      grid.push(row);
    }
    // March through grid cells, collect line segments where p crosses 0.5
    const {xScale, yScale} = makeScales();
    const segments = [];
    for (let iy = 0; iy < ny; iy++) {
      for (let ix = 0; ix < nx; ix++) {
        const tl = grid[iy][ix], tr = grid[iy][ix+1];
        const bl = grid[iy+1][ix], br = grid[iy+1][ix+1];
        const x0 = xLim[0] + ix * dx, x1 = x0 + dx;
        const y0 = yLim[0] + iy * dy, y1 = y0 + dy;
        const pts = [];
        if ((tl - 0.5) * (tr - 0.5) < 0) { const t = (0.5 - tl)/(tr - tl); pts.push([x0 + t*dx, y0]); }
        if ((tr - 0.5) * (br - 0.5) < 0) { const t = (0.5 - tr)/(br - tr); pts.push([x1, y0 + t*dy]); }
        if ((bl - 0.5) * (br - 0.5) < 0) { const t = (0.5 - bl)/(br - bl); pts.push([x0 + t*dx, y1]); }
        if ((tl - 0.5) * (bl - 0.5) < 0) { const t = (0.5 - tl)/(bl - tl); pts.push([x0, y0 + t*dy]); }
        if (pts.length >= 2) {
          segments.push([[xScale(pts[0][0]), yScale(pts[0][1])], [xScale(pts[1][0]), yScale(pts[1][1])]]);
          if (pts.length === 4) {
            segments.push([[xScale(pts[2][0]), yScale(pts[2][1])], [xScale(pts[3][0]), yScale(pts[3][1])]]);
          }
        }
      }
    }
    contourPaths = segments;
  }

  container.querySelectorAll(".mode-btn").forEach(btn => {
    btn.addEventListener("click", () => {
      mode = btn.dataset.mode;
      degree = 1;
      if (mode === "categorical") loadCategorical();
      else loadRegression();
      redraw();
    });
  });

  container.querySelectorAll(".deg-btn").forEach(btn => {
    btn.addEventListener("click", () => {
      degree = +btn.dataset.deg;
      redraw();
    });
  });

  container.querySelectorAll(".lam-btn").forEach(btn => {
    btn.addEventListener("click", () => {
      lambda_ = +btn.dataset.lam;
      redraw();
    });
  });

  container.querySelector("#fit-btn").addEventListener("click", () => {
    if (mode === "regression") fitRegression();
    else fitCategorical();
    redraw();
  });

  container.querySelector("#reset-btn").addEventListener("click", () => {
    if (mode === "categorical") loadCategorical();
    else loadRegression();
    degree = 1;
    redraw();
  });

  svg.on("click", (event) => {
    const [mx, my] = d3.pointer(event, g.node());
    if (mx < 0 || mx > innerW || my < 0 || my > innerH) return;
    const {xScale, yScale} = makeScales();
    const xVal = xScale.invert(mx);
    const yVal = yScale.invert(my);
    if (mode === "categorical") {
      const yCls = event.shiftKey ? 0 : 1;
      ptsX = [...ptsX, xVal];
      ptsX1 = [...ptsX1, yVal];
      ptsY = [...ptsY, yCls];
    } else {
      ptsX = [...ptsX, xVal];
      ptsY = [...ptsY, yVal];
    }
    fitLine = null;
    contourPaths = [];
    wFit = null;
    redraw();
  });

  redraw();
  return container;
}

Lab Summary

In this lab you:

• switched between regression and categorical overfitting examples
• fit models at different polynomial degrees and saw underfitting and overfitting
• added new points and observed how the fitted curve or decision boundary changes
• compared \(\lambda = 0\) and \(\lambda > 0\) at high degree and saw regularization smooth the fit
• connected the interactive tool to the three remedies from the lecture (more data, fewer features, regularization)

The regularization lab goes further. You will implement the regularized cost and gradient in Python and train on microchip quality-assurance data.

What Comes Next

You have seen how models can underfit (high bias) or overfit (high variance) in both regression and classification, and three practical responses when overfitting appears (more data, fewer features, regularization).

The next page formulates regularization precisely and shows how to apply it in linear and logistic regression.

If you want a second perspective first, the statistics course walks through overfitting with polynomial models and introduces the \(L_2\) penalty that regularization builds on.

Later, in Course 2, the bias and variance page returns to underfitting and overfitting with a numeric toolkit, \(J_{train}\), \(J_{cv}\), and learning curves, for diagnosing which one a model has.

Progress

91% completed (1481 of 1626 min)