Sequence Models

Course 5 of 5 in the Deep Learning Specialization by Andrew Ng (DeepLearning.AI).

Topics

Course 5 of 5 in the Deep Learning Specialization by Andrew Ng (DeepLearning.AI).

Notes

Recurrent Neural Networks
deep-learning
sequence-models
rnn
nlp
notation
forward-propagation
backpropagation

Sequence problems and their notation, why a standard network fails on them, and how a recurrent neural network runs forward and backward through time.

Aug 22, 2026

RNN Architectures and Language Models
deep-learning
sequence-models
rnn
nlp
language-model
sampling

The one-to-many, many-to-one, and encoder-decoder shapes an RNN can take, how a language model scores a sentence, and how to sample new text from one.

Aug 23, 2026

The word cat determines was much later in a sentence. Below it, backward arrows become thinner as they cross earlier RNN time steps.

Vanishing Gradients, GRUs, and LSTMs
deep-learning
sequence-models
rnn
gru
lstm
vanishing-gradients

Why a basic RNN cannot hold information across a long sentence, and how GRUs and LSTMs mitigate the problem.

Aug 24, 2026

Four inputs feed two recurrent chains. The forward chain runs from time step 1 to time step 4, while the backward chain runs from time step 4 to time step 1. Both hidden states at each time step feed its prediction.

Bidirectional and Deep RNNs
deep-learning
sequence-models
rnn
brnn
deep-rnn
architecture

Two extensions that finish the toolbox, one that lets a prediction see the future of the sequence and one that stacks recurrent layers into a deeper model.

Aug 24, 2026

Four stacks of sheets represent the input x, hidden state a, LSTM cell state c, and prediction y_pred. The front sheet of each stack is outlined and labeled as its two-dimensional slice at time step t.

Lab: Building Your Recurrent Neural Network Step by Step
deep-learning
sequence-models
rnn
lstm
backpropagation
numpy
lab

Implement the RNN cell, the LSTM cell, their forward passes over a sequence, and the optional backward passes through time, in plain NumPy.

Aug 24, 2026

Five columns of circles. The bottom row holds the inputs, a zero vector followed by the characters z, u, u and l. The middle row holds hidden states connected left to right by orange recurrence arrows. The top row holds the predictions z, u, u, l and newline, each one equal to the input of the following column.

Lab: Character Level Language Model
deep-learning
sequence-models
rnn
language-model
gradient-clipping
sampling
lstm
keras
lab

Train a character-level RNN on a list of dinosaur names, with gradient clipping and sampling written in NumPy, then generate poetry from a stacked LSTM.

Aug 24, 2026

Three columns, each showing a slice of X passing through a reshaper box, then an LSTM cell, then a dense softmax producing a prediction. Orange arrows carry the hidden and cell states from one column to the next. A bracket underneath marks that all columns share one set of weights.

Lab: Improvise a Jazz Solo with an LSTM Network
deep-learning
sequence-models
lstm
keras
functional-api
music-generation
lab

Train an LSTM on a jazz corpus using the Keras functional API with shared layers, then feed its own predictions back to generate and play a new solo.

Aug 25, 2026

A schematic two-dimensional scatter of fourteen labeled words. People words man, woman, king, and queen cluster together, animals dog, cat, and fish cluster together, numbers one, two, three, and four cluster together, and fruits grape, apple, and orange cluster together.

Introduction to Word Embeddings
deep-learning
sequence-models
nlp
word-embeddings
transfer-learning
cosine-similarity

Why one-hot vectors cannot generalize across words, how featurized word embeddings fix that, and how analogy reasoning and the embedding matrix work.

Aug 25, 2026

Six words each map to a one-hot vector, then through a shared matrix E to a 300-dimensional embedding. All six embeddings feed one hidden layer with parameters W superscript 1 and b superscript 1, which feeds a softmax layer with parameters W superscript 2 and b superscript 2, producing a distribution over 10,000 words.

Learning Word Embeddings with Word2vec and GloVe
deep-learning
sequence-models
nlp
word-embeddings
word2vec
skip-gram
negative-sampling
glove

Four algorithms that learn the embedding matrix, from a neural language model through skip-gram, negative sampling, and the GloVe co-occurrence objective.

Aug 25, 2026

Four words, The, dessert, is, and excellent, each map to a one-hot vector, then through the matrix E to a 300-dimensional embedding. All four embeddings feed an average box, whose 300-dimensional output feeds a softmax producing a rating from one to five stars.

Sentiment Classification and Debiasing Word Embeddings
deep-learning
sequence-models
nlp
word-embeddings
sentiment-classification
bias
fairness

Two applications of word embeddings, a sentiment classifier that works from a small labeled set, and a method for removing gender bias from the embeddings.

Aug 26, 2026

Three panels, each showing two arrows from a common origin with the angle between them marked. Left, father and mother at 27 degrees with cosine 0.89. Center, ball and crocodile at 74 degrees with cosine 0.27. Right, france minus paris and rome minus italy at 132 degrees with cosine minus 0.68.

Lab: Operations on Word Vectors
deep-learning
sequence-models
nlp
word-embeddings
glove
cosine-similarity
bias
fairness
numpy
lab

Load pretrained GloVe vectors, measure similarity with the cosine, solve word analogies, then neutralize biased words and equalize gender-specific pairs.

Aug 26, 2026

A grouped horizontal bar chart with five rows, one per emoji class. Each row has a training bar and a test bar. The smile class is the largest in both splits. Fork is the smallest in training, while heart and fork tie for smallest in the test split.

Lab: Emojify
deep-learning
sequence-models
nlp
word-embeddings
glove
lstm
keras
embedding-layer
sentiment-classification
lab

Build two emoji classifiers on 132 sentences, a softmax layer over averaged GloVe vectors and a two-layer LSTM with a frozen pretrained embedding.

Aug 26, 2026

Two chains of recurrent cells joined end to end. The left chain has five cells reading the five French words and no outputs. Its final activation feeds the first cell of the right chain, which has six cells each producing an English word, and each generated word is fed back as the input to the next cell.

Sequence to Sequence Models
deep-learning
sequence-models
nlp
machine-translation
encoder-decoder
image-captioning
conditional-language-model
greedy-search

The encoder-decoder architecture behind machine translation and image captioning, why it is a conditional language model, and why greedy search fails.

Aug 27, 2026

A tree growing left to right over three columns. The first column holds in, Jane and September. The second holds in September, Jane is and Jane visits, with orange arrows showing that two of them came from Jane and none from September. The third holds in September Jane, Jane is visiting and Jane visits Africa. Five faint stubs fan out of every node in the first two columns, standing in for the thousands of discarded alternatives.

Beam Search
deep-learning
sequence-models
nlp
machine-translation
beam-search
length-normalization
error-analysis
decoding

The approximate search that finds the most likely translation, the log and length-normalization refinements, and how to tell search errors from model errors.

Aug 27, 2026

A grouped bar chart with four groups for n equal to 1, 2, 3 and 4. Three bars per group. The repeated-the output scores 0.29 at n equals 1 and zero elsewhere. The cat output scores 0.71, 0.67, 0.40 and 0.25. The exact-match output scores 1.0 in every group.

BLEU Score
deep-learning
sequence-models
nlp
machine-translation
evaluation
bleu
n-grams
metrics

How to score a translation automatically when several different translations are equally good, using modified n-gram precision and a brevity penalty.

Aug 27, 2026

Two curves against sentence length in words. The encoder-decoder curve rises to a peak around twenty words and then declines steadily past forty words. The attention curve rises to a similar peak and stays roughly flat across the whole range.

Attention Model
deep-learning
sequence-models
nlp
machine-translation
attention
context-vector
bidirectional-rnn

Why one encoding vector fails on long sentences, and how letting the decoder build a fresh weighted view of the input at every step fixes it.

Aug 27, 2026

Two stacked panels sharing a time axis over a two second recording of speech. The upper panel plots the waveform as amplitude against time, showing four bursts of sound separated by quiet. The lower panel plots the spectrogram of the same clip, with frequency on the vertical axis up to 5000 hertz and brighter color where more energy sits. Each burst shows a stack of evenly spaced harmonic stripes in the low frequencies, brighter formant bands above them, and diffuse high frequency energy during the consonants.

Speech Recognition and Trigger Word Detection
deep-learning
sequence-models
speech-recognition
audio
spectrogram
ctc
attention
trigger-word-detection

How sequence to sequence models read audio, with spectrograms, the attention and CTC approaches to transcription, and a trigger word detector.

Aug 28, 2026

A left-to-right pipeline. The previous decoder state s of the previous step enters a RepeatVector block that copies it thirty times, then a Concatenate block joins it with the encoder states a. That feeds a dense layer with ten tanh units, then a dense layer with one relu unit producing the energies, then a softmax over the position axis producing the attention weights alpha, then a dot product with the encoder states producing the context vector.

Lab: Neural Machine Translation with Attention
deep-learning
sequence-models
nlp
attention
machine-translation
keras
lstm
lab

Build an attention model in Keras that translates human-readable dates into the ISO format, then plot what each output character attends to.

Aug 29, 2026

Two stacked panels. The upper panel plots the background audio waveform, amplitude against time over ten seconds. The lower panel plots its spectrogram, with frequency on the vertical axis and time on the horizontal, showing broad low frequency energy across the whole clip.

Lab: Trigger Word Detection
deep-learning
sequence-models
speech-recognition
audio
spectrogram
gru
conv1d
keras
lab

Synthesize a speech dataset from scratch, then run a convolutional and recurrent model that detects the word activate in a ten second clip.

Aug 29, 2026

Three panels under one long arrow labeled increased complexity, still one token at a time. The left panel, RNN, holds a single cyan tanh circle with the previous activation entering from the left and the new activation leaving to the right. The middle panel, GRU, holds a green reset gate feeding a cyan tanh candidate, a green update gate, and a purple combine box sitting on the orange state line. The right panel, LSTM, holds three green gate circles and one cyan tanh candidate on a lower row, a purple multiply and a purple add on the orange cell-state line above them, and a second cyan tanh with two further purple multiply nodes on an intermediate row between the two.

Transformer Network
deep-learning
sequence-models
nlp
transformer
self-attention
multi-head-attention
positional-encoding
machine-translation

Self-attention, multi-head attention, and how encoder and decoder blocks, positional encoding, residual connections, and masking combine into the transformer.

Aug 30, 2026

A heatmap 512 columns wide and 50 rows tall in a red to blue diverging colormap. On the left, broad bands of red and blue curve and interleave down the rows, the pattern repeating more tightly toward the right. Beyond roughly column 200 the image becomes a fine vertical pinstripe of alternating near-neutral and saturated columns, the sine dimensions sitting near zero and the cosine dimensions near one, and those columns barely change from the top row to the bottom.

Lab: Building a Transformer Network
deep-learning
sequence-models
nlp
transformer
self-attention
multi-head-attention
positional-encoding
masking
keras
glove
lab

Implement positional encoding, masking, scaled dot-product attention and the encoder and decoder stacks, then measure what position does to word embeddings.

Aug 31, 2026

Two panels. The left is a scatter plot with a logarithmic horizontal axis labeled test support and a vertical axis from 0 to 1 labeled F1 score, holding twelve labeled points. Three sit flat on zero at the left end, the rest scatter between roughly 0.5 and 0.95 with no tight trend, and the highest-support point is also the highest scoring. The right is a horizontal grouped bar chart with two bars per category, one for the true subtoken count and one for the predicted count, on a logarithmic axis. Bars representing a count of zero are drawn as short stubs labeled zero, since a logarithmic axis cannot show a zero, and several categories carry such a stub on the predicted bar.

Lab: Fine-Tuning a Pretrained Transformer
deep-learning
sequence-models
nlp
transformer
transfer-learning
fine-tuning
huggingface
distilbert
pytorch
named-entity-recognition
question-answering
lab

Fine-tune DistilBERT for entity recognition on resumes and extractive question answering, and fix the label alignment bugs behind the original results.

Aug 31, 2026
No matching items
Back to top