Improving Deep Neural Networks

regularization

dropout

L2 and L1 regularization, weight decay, dropout and inverted dropout, plus data augmentation and early stopping to reduce overfitting.

Aug 3, 2026

Setting Up Your Optimization Problem

Input normalization, vanishing and exploding gradients, careful weight initialization, and gradient checking to verify backpropagation.

Lab: Initialization

lab

initialization

Train the same 3-layer network with zeros, large random, and He initialization and compare convergence, accuracy, and decision boundaries.

Lab: Regularization

Implement L2 regularization and inverted dropout by hand, then compare a baseline, an L2 model, and a dropout model on a 2D football dataset.

Lab: Gradient Checking

lab

gradient-checking

Verify backpropagation with gradient checking, from a 1D linear model to a full 3-layer network, and hunt down two planted backprop bugs.

Mini-batch Gradient Descent

Splitting the training set into mini-batches, the curly-brace notation, epochs, cost curve noise, and choosing a mini-batch size.

Exponentially Weighted Averages

Exponentially weighted moving averages, the effect of beta, why the formula averages over 1/(1-beta) days, and bias correction.

Momentum, RMSprop, and Adam

Gradient descent with momentum, RMSprop, the Adam optimizer, learning rate decay schedules, and why saddle points beat local optima.

Lab: Optimization Methods

Implement mini-batch creation, momentum, and Adam by hand, then train a 3-layer network on the moons dataset with learning rate decay schedules.

Hyperparameter Tuning

hyperparameter-tuning

random-search

log-scale

Which hyperparameters to tune first, random search versus grid search, coarse to fine sampling, log scales, and panda versus caviar tuning.

Batch Normalization

How batch norm normalizes hidden unit values with learnable mean and variance, fits into training, why it works, and what changes at test time.

Softmax Regression

multi-class-classification

softmax

cross-entropy

How softmax generalizes logistic regression to C classes, with a worked example, linear decision boundaries, cross-entropy loss, and backprop.

Deep Learning Frameworks and TensorFlow

How to choose a deep learning framework, and the basic structure of a TensorFlow program with variables, GradientTape, and optimizers.