Case Study: Bird Recognition in the City of Peacetopia
Pilots train in flight simulators, where they practice many different flying scenarios safely and in a short amount of time. That gives them valuable experience far faster than flying real planes ever could.
This case study works the same way. It presents one machine learning project as a running story and asks you to make the decisions a real team faces, so you can practice the judgment calls that might otherwise take years on the job to encounter. The scenario below is adapted from a real production application, with details modified for confidentiality.
Problem Statement
You are a researcher in the city of Peacetopia. The residents of Peacetopia share a unique characteristic, which is that they are afraid of birds. To protect them, you are tasked with developing an algorithm that will detect any bird flying over Peacetopia and alert the population.
The City Council provides you with a dataset of 10,000,000 images of the sky above Peacetopia, captured by the city security cameras. They are labeled as follows.
- \(y = 0\), there is no bird in the image
- \(y = 1\), there is a bird in the image
Your goal is to create an algorithm capable of classifying new images taken by the security cameras in Peacetopia. You have several decisions to make regarding the evaluation metric and how to structure your data into train, dev, and test sets.
Choosing a Metric
The City Council specifies that they want an algorithm that has high accuracy, that operates quickly and takes only a short time to classify a new image, and that requires minimal memory so it can run on a small processor attached to the various security cameras.
The ideas behind this first group of questions come from Setting Up Your Goal.
Review Questions
1. True or False: You discuss with them the need for a singular evaluation metric to guide development.
True
False
a. True. Three separate numbers give you no way to rank two candidate models, so the team cannot quickly tell whether a new idea helped or hurt, and every comparison turns into a debate. Agreeing on one number to optimize before development starts is what keeps the iteration loop fast. Here that means treating accuracy as the optimizing metric and turning runtime and memory into satisficing thresholds.
1. The city revises its criteria to “we need an algorithm that can let us know a bird is flying over Peacetopia as accurately as possible”, “we want the trained model to take no more than 10 seconds to classify a new image”, and “we want the model to fit in 10MB of memory”. Given models with different accuracies, runtimes, and memory sizes, how would you choose one?
Accuracy is an optimizing metric, therefore the most accurate model is the best choice.
Take the model with the smallest runtime because that will provide the most overhead to increase accuracy.
Find the subset of models that meet the runtime and memory criteria. Then, choose the highest accuracy.
Create one metric by combining the three metrics and choose the best performing model.
c. The revised wording turns runtime and memory into satisficing metrics, since the council asks only that they stay under 10 seconds and 10MB rather than asking for them to be as small as possible. Accuracy is the optimizing metric. So you first discard every model that violates a threshold, then pick the most accurate survivor. Option a ignores the two constraints, and option d would let a small accuracy gain buy its way past a hard limit the hardware cannot exceed.
1. True or False: The essential difference between an optimizing metric and satisficing metrics is the priority assigned by the stakeholders.
False
True
a. False. The difference lies in the nature of the metrics themselves rather than in stakeholder priority. An optimizing metric is unbounded, meaning you want it as good as you can possibly get it and there is no point at which you stop caring. A satisficing metric has a specific threshold, and once that threshold is met, further improvement buys you nothing. Accuracy has no natural stopping point, while a 10 second runtime limit does, and that structural difference is what puts each metric in its role. Stakeholders do influence which quantity goes where, but priority is not what separates the two kinds.
Sizing the Splits
With the metric settled, the next decision is how to divide 10,000,000 images into training, dev, and test sets, and what to do when more data arrives later.
Review Questions
1. You propose a 95% / 2.5% / 2.5% split for train / dev / test to the City Council. They ask for your reasoning. Which of the following best justifies your proposal, given that the total data set contains 10,000,000 data points?
The emphasis on the training set provides the most accurate model, supporting the memory and processing efficiency.
The emphasis on the training set will allow us to iterate faster.
The most important goal is achieving the highest accuracy, and that can be done by allocating the maximum amount of data to the training set.
With a dataset comprising 10,000,000 individual samples, 2.5% represents 250,000 samples, which should be more than enough for dev and testing to evaluate bias and variance.
d. The old 60/20/20 rule of thumb was built for datasets of a few thousand examples. What actually matters is that the dev set is large enough to distinguish between the algorithms you are trying, and that the test set is large enough to give high confidence in the final system. At 250,000 images each, both are comfortably past that point, so everything left over is better spent on training. The justification is about absolute counts, not about percentages.
1. Now that you have set up your train, dev, and test sets, the City Council comes across another 1,000,000 images from social media and offers them to you. These images have a different distribution from the images the City Council originally provided, but you think they could help your algorithm. Should you add this data to the training set?
No.
Yes.
b. Yes. The training set does not have to match the distribution you care about, so extra data is usually worth having even when it comes from somewhere else. What must match is the dev and test sets, which have to be drawn from the security camera images that the deployed system will actually see. So add the social media images to the training set and leave dev and test untouched. Mixing them into dev or test would move the target away from the real problem.
1. One member of the City Council wants to add 1,000,000 citizen data images to the development (dev) set. Your original data is from security cameras, and you object to adding the citizen data because of which of the following? Choose all that apply.
The 1,000,000 citizen data images do not have a consistent input-output relationship as the security camera data.
This would cause the dev and test set distributions to become different. This is a bad idea because you are not aiming where you want to hit.
A bigger test set will slow down the speed of iterating because of the computational expense of evaluating models on the test set.
The dev set no longer reflects the distribution of data (security cameras) you most care about.
b and d. This is the same 1,000,000 extra images as the previous question, and the answer flips purely because of which set they go into. Dev and test have to come from the same distribution, and that distribution has to be the one you actually care about. Adding citizen photos to the dev set alone breaks both conditions at once, so the team would spend months tuning toward a target that does not match what the deployed cameras see. Option a claims the labels mean something different in the two sources, which is not true, since a bird is a bird in either kind of photo. Option c talks about the test set, which is not what was proposed, and evaluation cost is not the reason to object anyway.
Bias, Variance, and Human-Level Performance
Training is under way and the numbers start coming in. This group of questions comes from Comparing to Human-Level Performance.
Review Questions
1. You train a system and its errors are as follows, where error = 100% - accuracy.
| Training set error | 4.0% |
| Dev set error | 4.5% |
This suggests that one good avenue for improving performance is to train a bigger network to reduce the 4.0% training error. Do you agree?
No, because this shows your variance is higher than your bias.
Yes, because having a 4.0% training error shows you have a high bias.
No, because there is insufficient information to tell.
Yes, because this shows your bias is higher than your variance.
c. A 4.0% training error means nothing on its own, because bias is measured against Bayes error rather than against zero. If human-level performance is 0.5%, the avoidable bias is 3.5% and a bigger network is exactly right. If human-level performance is 4.0%, the avoidable bias is zero and the whole 0.5% gap to the dev set is a variance problem. Without an estimate of Bayes error you cannot tell which situation you are in.
1. If your goal is to use “human-level performance” as an estimate for Bayes error in a bird species identification task, how would you define “human-level performance”?
The performance of volunteer amateur ornithologists.
The best performance of a specialist (ornithologist) or a group of specialists.
The performance of the head of the City Council.
The performance of the average citizen.
b. Bayes error is the best error any mapping from image to label could achieve, so the proxy for it has to be the best error humans can achieve, which is a team of experienced specialists discussing and debating the hard cases. Amateurs, average citizens, and council members all sit well above that, and using one of them would make you believe you have less avoidable bias than you really do. A looser proxy hides real opportunity.
1. Which of the below shows the optimal order of accuracy from worst to best?
The learning algorithm’s performance -> Bayes error -> human-level performance.
Human-level performance -> Bayes error -> the learning algorithm’s performance.
Human-level performance -> the learning algorithm’s performance -> Bayes error.
The learning algorithm’s performance -> human-level performance -> Bayes error.
c. Bayes error is the ceiling that nothing can pass, so it is the most accurate entry in any ordering and belongs at the “best” end. The word optimal is what settles the other two. In the best case the algorithm has already surpassed human-level performance, which is possible and happens often, so human-level performance is the least accurate of the three. The ordering worst to best is therefore human-level performance, then the learning algorithm, then Bayes error. Option d describes a system that is still catching up to people, which is the common situation but not the optimal one.
1. Which of the following best describes the most effective next step in your project, given human-level performance of 0.1%, training set error of 2.0%, and dev set error of 2.1%?
Continue tuning until the training set error matches human-level performance, focusing solely on the optimizing metric.
Prioritize actions to decrease bias by increasing model complexity, as the training error significantly exceeds human-level performance.
Deploy the model to target devices to evaluate against satisficing metrics.
Evaluate the test set to determine the variance.
b. The avoidable bias is 2.0% minus 0.1%, which is 1.9%, while the variance is only 2.1% minus 2.0%, which is 0.1%. Avoidable bias is nineteen times larger, so the effort belongs there. Bias tactics include a bigger model, longer training or a better optimization algorithm, and a search over architectures and hyperparameters. Variance work such as regularization or more data would be attacking the smaller of the two gaps.
1. You have now also run your model on the test set and find that the error rate is 7.0% compared to a 2.1% error rate for the dev set. What should you do? Choose all that apply.
Increase the size of the dev set.
Try decreasing regularization for better generalization with the dev set.
Get a bigger test set to increase its accuracy.
Try increasing regularization to reduce overfitting to the dev set.
a and d. A large gap between dev error and test error means you have overfit to the dev set, which happens naturally because every hyperparameter decision was made by looking at that set. A bigger dev set makes it harder to overfit and gives a more reliable signal, and more regularization reduces the overfitting directly. Option b would make the overfitting worse. Option c does not help, because the test set was never tuned against, so its size is not the problem.
1. After working on this project for a year, you finally achieve the following. What can you conclude? Check all that apply.
| Human-level performance | 0.10% |
| Training set error | 0.05% |
| Dev set error | 0.05% |
With only 0.05% further progress to make, you should quickly be able to close the remaining gap to 0%.
If the test set is big enough for the 0.05% error estimate to be accurate, this implies Bayes error is \(\le 0.05\%\).
It is highly unlikely this result is purely a statistical anomaly, but statistical noise may still contribute to the error.
It is now harder to measure avoidable bias, thus progress will be slower going forward.
b and d. Achieving 0.05% error proves that error of 0.05% is attainable, so Bayes error can be no larger than that, which is what makes b correct. And now that training error has passed the human-level estimate of 0.10%, that estimate is no longer usable as a proxy for Bayes error. You cannot tell whether the true Bayes error is 0.05%, 0.03%, or 0.01%, so you cannot tell how much avoidable bias is left or whether to work on bias or variance, which is what makes d correct. Option a assumes Bayes error is zero, and nothing here says it is. Option c states the opposite of what the numbers support, since a matching 0.05% on both training and dev sets is a consistent result rather than an unlikely one.
Changing the Target and Keeping Iteration Fast
The system is working, and the remaining decisions are about what to do when the world moves out from under your metric.
Review Questions
1. Your system is now very accurate but has a higher false negative rate than the City Council of Peacetopia would like. What is your best next step?
Reset your “target” (metric) for the team and tune to it.
Pick false negative rate as the new metric, and use this new metric to drive all further development.
Look at all the models you have developed during the development process and find the one with the lowest false negative error rate.
Expand your model size to account for more corner cases.
a. A missed bird costs the residents far more than a false alarm does, so the current metric no longer reflects what the council actually cares about. When your metric and your dev set stop ranking systems the way you want them ranked, that is the signal to move the target and then aim at the new one. In practice that means putting a heavier weight on false negatives in the error calculation. Option b throws away accuracy altogether, and option c picks a winner using a metric that was never optimized for.
1. You have handily beaten your competitor, and your system is now deployed in Peacetopia and is protecting the citizens from birds. But over the last few months, a new species of bird has been slowly migrating into the area, so the performance of your system slowly degrades because your data is being tested on a new type of data. You have only 1,000 images of the new species of bird, and the city expects a better system from you within the next 3 months. Which of these should you do first?
Use the data you have to define a new evaluation metric (using a new dev/test set) that accounts for the new species, and use that metric to guide further improvements.
Try data augmentation or data synthesis to get more images of the new type of bird.
Put the 1,000 images into the training set so as to try to do better on these birds.
Add the 1,000 images into your dataset and reshuffle into a new train/dev/test split.
a. The distribution you care about has changed, so the first move is to point the target at it. Until the dev and test sets contain the new species, you have no way to measure whether anything you try afterwards actually helps. Options b and c are reasonable things to do later, but doing them first leaves you improving a number that no longer describes the problem. Option d would mix the new distribution into all three sets and still leave the dev and test sets dominated by the old one.
1. The City Council thinks that having more cats in the city would help scare off birds. They are so happy with your work on the bird detector that they also hire you to build a cat detector. Because of years of working on cat detectors, you have such a huge dataset of 100,000,000 cat images that training on this data takes about two weeks. Which of the statements do you agree with? Check all that agree.
Needing two weeks to train will limit the speed at which you can iterate.
Buying faster computers could speed up your team’s iteration speed and thus your team’s productivity.
If 100,000,000 examples is enough to build a good enough cat detector, you might be better off training with just 10,000,000 examples to gain a roughly 10x improvement in how quickly you can run experiments, even if each model performs a bit worse because it is trained on less data.
Having built a good bird detector, you should be able to take the same model and hyperparameters and just apply it to the cat dataset, so there is no need to iterate.
a, b, and c. Applied machine learning is an empirical loop of idea, code, and experiment, so anything that shortens one lap through the loop buys you more ideas tested per month. A two week training run is a hard cap on that. Faster hardware and a smaller subset of the data both attack the same bottleneck, and trading a little final accuracy for ten times the experiments is usually a good trade while you are still exploring. Option d is wrong because a good architecture for one task rarely transfers unchanged to another, and the only way to find out is to iterate.