Case Study: Autonomous Driving
This is the second machine learning flight simulator of the course. Like Bird Recognition in the City of Peacetopia, it presents one project as a running story and asks you to make the decisions a real team faces. The first case study exercised metric choice and data splits. This one exercises the later material, meaning error analysis, handling mismatched training and dev/test distributions, transfer and multi-task learning, and the choice between a pipeline and an end-to-end system.
Problem Statement
You are employed by a startup building self-driving cars. You are in charge of detecting road signs (stop sign, pedestrian crossing sign, construction ahead sign) and traffic signals (red and green lights) in images. The goal is to recognize which of these objects appear in each image. For example, an image showing a pedestrian crossing sign and red traffic lights would be labeled
\[ y^{(i)} = \begin{bmatrix} 0 \\ 1 \\ 0 \\ 1 \\ 0 \end{bmatrix} \begin{array}{l} \text{stop sign} \\ \text{pedestrian crossing sign} \\ \text{construction ahead sign} \\ \text{red traffic light} \\ \text{green traffic light} \end{array} \]
Your 100,000 labeled images are taken using the front-facing camera of your car. This is also the distribution of data you care most about doing well on. You think you might be able to get a much larger dataset off the internet, which could be helpful for training even if the distribution of internet data is not the same.
Getting Started
The ideas behind this first group of questions come from Build Your First System Quickly, Then Iterate, Carrying Out Error Analysis, and Multi-task Learning.
Review Questions
1. You are getting started with this project. What is the first thing you do? Assume each of the steps below would take about an equal amount of time (a few days).
Spend some time searching the internet for the data most similar to the conditions you expect on production.
Train a basic model and do error analysis.
Spend a few days collecting more data using the front-facing camera of your car, to better understand how much data per unit time you can collect.
Invest a few days in thinking on potential difficulties, and then some more days brainstorming about possible solutions, before training any model.
b. Applied machine learning is an iterative process, and the recommendation for a brand new application is to build a first system quickly and then iterate. A basic model plus error analysis replaces speculation with evidence, because the mistakes the model actually makes tell you whether internet data, more camera data, or something else entirely deserves the next few days. Options a, c, and d all spend days acting on guesses about what will matter before a single experiment has produced any signal.
1. Your goal is to detect road signs and traffic signals in images, and you plan to use a deep neural network with ReLU units in the hidden layers. Suppose that you use a sigmoid function for the output layer, and the output \(\hat{y}\) has shape (5, 1). Which of the following best describes the cost function?
\(\displaystyle \frac{1}{m} \sum_{i=1}^{m} \left( -y^{(i)} \log \hat{y}^{(i)} - (1 - y^{(i)}) \log(1 - \hat{y}^{(i)}) \right)\)
\(\displaystyle \frac{1}{m} \sum_{i=1}^{m} \sum_{j=1}^{5} \mathcal{L}(\hat{y}_j^{(i)}, y_j^{(i)})\)
\(\displaystyle \frac{1}{m} \sum_{i=1}^{m} \sum_{j=1}^{5} \mathcal{L}(\hat{y}_i^{(j)}, y_i^{(j)})\)
\(\displaystyle \frac{\exp \hat{y}_j^{(i)}}{\sum_{j=1}^{5} \exp \hat{y}_j^{(i)}}\)
b. This is multi-task learning. One image can contain several of the five objects at once, so the network has five sigmoid output units, and the cost sums the logistic loss over the five components of each label vector and then averages over the \(m\) training examples. The superscript \((i)\) indexes training examples and the subscript \(j\) indexes the five objects, which is exactly what option b writes. Option a is the cost for a single binary output. Option c swaps the two indices, so it sums over five training examples and averages over objects, which is not a meaningful quantity. Option d is a softmax activation rather than a cost function, and softmax would be wrong here anyway because it forces exactly one class per image while these images can contain several objects.
1. You are carrying out error analysis and counting up what errors the algorithm makes. Which of these datasets do you think you should manually go through and carefully examine, one image at a time?
10,000 randomly chosen images
10,000 images on which the algorithm made a mistake
500 images on which the algorithm made a mistake
500 randomly chosen images
c. Error analysis means looking at the examples the algorithm got wrong, because correctly classified images tell you nothing about what to fix. That rules out the two random samples, which would be dominated by correct predictions. Between the remaining two, around 100 to 500 mistakes is enough to estimate what fraction of errors falls into each category and therefore where the ceilings are, while hand-examining 10,000 mistakes costs twenty times the effort for very little extra precision. Keep the manual pass small enough to actually finish.
Working with Two Data Sources
After working on the data for several weeks, your team ends up with 100,000 labeled images taken from the front-facing camera and 900,000 labeled images of roads downloaded from the internet. Each image’s labels precisely indicate the presence of any specific road signs and traffic signals or combinations of them, so for example \(y^{(i)} = \begin{bmatrix} 1 & 0 & 0 & 1 & 0 \end{bmatrix}^T\) means the image contains a stop sign and a red traffic light. These questions come from Partially Labeled Data and Training and Testing on Different Distributions.
Review Questions
1. True or False: Because this is a multi-task learning problem, when an image is not fully labeled, for example \(y^{(i)} = \begin{bmatrix} 0 & ? & ? & 1 & 0 \end{bmatrix}^T\), you can use it if you ignore those entries when calculating the loss function.
False
True
b. True. With one sigmoid unit per label, the inner sum of the cost only needs to run over the components that actually have a 0 or 1 label, skipping the question marks. A partially labeled image still contributes gradient signal for the objects that were labeled, so the data remains usable. This is one of the practical conveniences of the multi-task setup.
1. True or False: The distribution of data you care about contains images from your car’s front-facing camera, which comes from a different distribution than the images you were able to find and download off the internet. The best way to split the data is using the 900,000 internet images to train, and divide the 100,000 images from your car’s front-facing camera between dev and test sets.
False
True
a. False. Dev and test must indeed contain only front-facing camera images, but 50,000 examples each is far more than an evaluation needs, and every camera image parked in dev or test is one the training set never benefits from. A better split moves most of the camera images into training, for example 900,000 internet plus 80,000 camera images to train on, with 10,000 camera images each for dev and test. That keeps the target on the camera distribution while letting the algorithm see a substantial amount of data from the distribution it will be judged on, exactly as in the speech activated rearview mirror example.
Diagnosing Bias, Variance, and Data Mismatch
The splits are chosen and the error numbers start coming in. These questions come from Bias and Variance with Mismatched Data Distributions.
Review Questions
1. Assume you have finally chosen the following split between the data. You also know that human-level error on the road sign and traffic signals classification task is around 0.5%. Which of the following is true?
| Dataset | Contains | Error of the algorithm |
|---|---|---|
| Training | 940,000 images randomly picked from (900,000 internet images + 60,000 car’s front-facing camera images) | 1% |
| Training-dev | 20,000 images randomly picked from (900,000 internet images + 60,000 car’s front-facing camera images) | 5.1% |
| Dev | 20,000 images from your car’s front-facing camera | 5.6% |
| Test | 20,000 images from the car’s front-facing camera | 6.8% |
The size of the train-dev set is too large.
You have a high bias.
You have a large data-mismatch problem.
You have a high variance problem.
d. Walk down the list of gaps. Avoidable bias is training error minus human-level error, 1% minus 0.5%, which is 0.5% and small. Variance is training-dev error minus training error, 5.1% minus 1%, which is 4.1% and by far the largest gap. Data mismatch is dev error minus training-dev error, 5.6% minus 5.1%, which is 0.5% and small. The error jumps on data from the very same distribution the network trained on but never saw, so the network is failing to generalize, which is a variance problem. Option a is wrong because 20,000 examples is a perfectly reasonable training-dev size, and options b and c point at the two small gaps.
1. Assume you have finally chosen the following split between the data. Human-level error on this task is approximately 0.5%, and human-level error is a good estimation of Bayes error. True or False: Based on this, the Bayes error for the car camera images (dev/test) is higher than the Bayes error for the mixed internet/car images (training).
| Dataset | Contains | Error of the algorithm |
|---|---|---|
| Training | 940,000 images randomly picked from (900,000 internet images + 60,000 car’s front-facing camera images) | 2% |
| Training-dev | 20,000 images randomly picked from (900,000 internet images + 60,000 car’s front-facing camera images) | 2.3% |
| Dev | 20,000 images from your car’s front-facing camera | 1.3% |
| Test | 20,000 images from the car’s front-facing camera | 1.1% |
True
False
b. False. The first two rows are measured on the mixed distribution and the last two on the camera distribution, and the numbers go down when you cross that boundary, from 2.3% on training-dev to 1.3% on dev and 1.1% on test. The algorithm performs better on the camera images than on the data it was trained on, which suggests the camera images are the easier distribution. That is evidence that their Bayes error is, if anything, lower than for the mixed internet and car images, not higher. This is the situation from the more general formulation, where the dev and test distribution happens to be easier and the usual always-increasing pattern of errors reverses.
Error Analysis on the Dev Set
You decide to focus on the dev set and check by hand what the errors are due to. Here is a table summarizing your discoveries.
| Overall dev set error | 15.3% |
| Errors due to incorrectly labeled data | 4.1% |
| Errors due to foggy pictures | 8.0% |
| Errors due to rain drops stuck on your car’s front-facing camera | 2.2% |
| Errors due to other causes | 1.0% |
In this table, 4.1%, 8.0%, and so on are a fraction of the total dev set, not just of the examples your algorithm mislabeled. For example, about 8.0/15.3 = 52% of your errors are due to foggy pictures. These questions come from Carrying Out Error Analysis, Addressing Data Mismatch, and Cleaning Up Incorrectly Labeled Data.
Review Questions
1. True or False: Should the team’s highest priority be to bring more foggy pictures into the training set to address the 8.0% of errors in that category?
True because it is the largest category of errors. We should always prioritize the largest category of errors as this will make the best use of the team’s time.
First start with the sources of error that are least costly to fix.
False because it depends on how easy it is to add foggy data. If foggy data is very hard and costly to collect, it might not be worth the team’s effort.
True because it is greater than the other error categories added together (8.0 > 4.1 + 2.2 + 1.0).
c. The 8.0% figure is a ceiling, meaning the most that perfect performance on foggy images could improve the overall error. A ceiling tells you the potential value of a fix, but choosing what to work on weighs that value against the cost and feasibility of the fix, and collecting or synthesizing enough foggy data might be very expensive. Options a and d turn the largest category into an automatic winner, which ignores that trade-off entirely, and option b makes the opposite mistake by ranking on cost alone while ignoring how much each fix could possibly buy.
1. You can buy a specially designed windshield wiper that helps wipe off some of the raindrops on the front-facing camera. Which one of the following statements do you agree with?
2.2% would be a reasonable estimate of the maximum amount this windshield wiper could improve performance.
2.2% would be a reasonable estimate of the minimum amount this windshield wiper could improve performance.
2.2% would be a reasonable estimate of how much this windshield wiper could worsen performance in the worst case.
2.2% would be a reasonable estimate of how much this windshield wiper will improve performance.
a. Raindrop errors account for 2.2% of the dev set, so even a wiper that removed every raindrop perfectly could reduce the overall error by at most 2.2 percentage points. The realistic gain is smaller, since the wiper only wipes off some of the drops and some raindrop images might be misclassified for other reasons too. That makes 2.2% a ceiling, not a guarantee, which is why options b and d overpromise. Option c is about a risk the table says nothing about.
1. You decide to use data augmentation to address foggy images. You find 1,000 pictures of fog off the internet and “add” them to clean images to synthesize foggy days. Which one of the following do you agree with?
With this technique, we duplicate the size of the training set by synthesizing a new foggy image for each image in the training set.
If used, the synthetic data should be added to the training/dev/test sets in equal proportions.
If used, the synthetic data should be added to the training set.
It is irrelevant how the resulting foggy images are perceived by the human eye; the most important thing is that they are correctly synthesized.
c. Artificial data synthesis is a training set technique. The dev and test sets define the target and must contain only real images from the front-facing camera, so synthetic images have no place there, which also rules out option b. Option d has it backwards, because if the synthesized fog does not look like real fog to a person, there is little reason to expect it to match what the camera will see, and the whole point is to move the training data closer to the real distribution. Option a describes something the technique does not require, and pasting the same 1,000 fog patterns across the entire training set would risk the small subset problem, where the network overfits to that tiny slice of all possible fog.
1. After working further on the problem, you have decided to correct the incorrectly labeled data. Your team corrects the labels of the wrongly predicted images on the dev set. True or False: You need to correct the labels of the test set so that the test and dev sets have the same distribution, but you will not change the labels on the train set because most models are robust enough that they are not severely affected by the difference in distributions.
False, the test set should not be changed since we want to know how the model performs with uncorrected or original data.
True, as pointed out, we must keep dev and test with the same distribution. The labels in the training set should be fixed only in case of a systematic error.
False, the test set should be changed, but also the train set to keep the same distribution between the train, dev, and test sets.
b. Whatever label correction process you apply to the dev set must be applied to the test set as well, because the two sets have to keep coming from the same distribution or the target splits in two. The training set is different. Learning algorithms are quite robust to random label errors in a large training set, and this page has already established that training may come from a different distribution anyway, so retouching a million training labels is rarely worth the effort. The exception is systematic errors, such as a labeler consistently marking green lights as red, which the algorithm will learn as if they were true.
Transfer Learning, Multi-Task Learning, and End-to-End
The detector works, and now colleagues start asking how your work and your data can help their projects. These questions come from Transfer Learning, Multi-task Learning, and Whether to Use End-to-End Deep Learning.
Review Questions
1. One of your colleagues at the startup is starting a project to classify road signs as stop, dangerous curve, construction ahead, dead-end, and speed limit signs. Given how specific the signs are, your colleague has only a small dataset and has not been able to create a good model. You offer your help providing the trained weights (parameters) of your model to transfer knowledge. True or False: Your colleague points out that this problem has more specific items than the ones you used to train your model. This makes the transfer of knowledge impossible.
False
True
a. False. This is the classic setting where transfer learning shines. Both tasks take the same input type, road images, you have a lot of data for the source task, and your colleague has little data for the target task. The early layers of your network have already learned low level image features such as edges, shapes, and sign-like textures from a large dataset, and those features are just as useful for distinguishing dead-end signs from speed limit signs. Your colleague deletes your output layer, adds a new one for the five sign classes, and fine-tunes on the small dataset. The output labels being different is not an obstacle, it is the normal case for transfer learning.
1. Another colleague wants to use microphones placed outside the car to better hear if there are other vehicles around you. For example, if there is a police vehicle behind you, you would be able to hear their siren. However, your colleague does not have much data to train this audio system. How can you help?
Multi-task learning from your vision dataset could help your colleague get going faster. Transfer learning seems significantly less promising.
Either transfer learning or multi-task learning could help our colleague get going faster.
Transfer learning from your vision dataset could help your colleague get going faster. Multi-task learning seems significantly less promising.
Neither transfer learning nor multi-task learning seems promising.
d. Both techniques need the tasks to share an input. Transfer learning helps when the source and target tasks have the same input type, but the low level features a network learns from camera images say nothing useful about audio waveforms. Multi-task learning trains a single network on one shared input \(x\) with several outputs, and an image and an audio clip cannot be the same \(x\). However much goodwill there is between the two projects, a vision dataset simply has nothing to transfer to a sound problem.
1. You are building a system to recognize stop signs. Your approach is that first, a neural network predicts bounding box coordinates around potential traffic signs in an image, and second, a separate neural network determines if each predicted sign is a stop sign. Is this an example of multi-task learning?
False
True
a. False. Multi-task learning means one network, trained on one input, producing several outputs simultaneously, like the five-component label vector at the top of this page. What is described here is two networks in a pipeline, where the output of a localization network becomes the input of a classification network. Each network does exactly one task. That is a hand-designed multi-step pipeline, which is a perfectly reasonable architecture, but it is the opposite of folding several tasks into a single network.
1. To recognize a stop sign, you use the following approach. First, localize any traffic sign in an image. After that, determine if the sign is a stop sign or not. This is a better approach than an end-to-end model for which of the following cases? Choose the best answer.
There is not enough data to train a big neural network.
There is a large amount of data.
The problem has a high Bayes error.
There are available models which we can use to transfer knowledge.
a. The key question for end-to-end deep learning is whether you have enough data to learn the full function mapping directly from image to stop-sign-or-not. When you do not, breaking the problem into two simpler subtasks wins, because each subtask needs a less complex function and each usually has more data available, such as datasets of localized signs and datasets of cropped signs to classify. Option b is precisely the case where end-to-end starts to shine and the pipeline loses its advantage. Bayes error and the availability of pretrained models are separate concerns that do not decide between the two designs.