Bayes Theorem

probability

Imagine the following scenario. There is a rare disease going around and you want to get tested for it. You go to the doctor and the doctor says: “I…

Published

June 20, 2026

Medical Test Example

Imagine the following scenario. There is a rare disease going around and you want to get tested for it. You go to the doctor and the doctor says: “I have a very effective test, which is correct most of the time.” You take the test, go home, and then the doctor calls to say you tested positive.

Maybe it is time to panic. But before panicking, let us do some math.

The question is: what is the probability that you actually have the disease, given that you tested positive? This is an example of Bayes theorem, one of the most important results in probability. It is used all over the place, including in machine learning (spam detection, speech recognition, and many other applications).

Setting Up the Numbers

Suppose there are 1,000,000 people in the population, and the illness affects only 1 in every 10,000 people.

It means two things:

Out of every 100 sick people, 99 get correctly diagnosed as sick, and 1 gets incorrectly diagnosed as healthy.
Out of every 100 healthy people, 1 gets misdiagnosed as sick, and 99 get correctly diagnosed as healthy.

You went to the doctor and tested positive (diagnosed sick). The question is: are you really sick? More specifically, what is \(P(\text{sick} \mid \text{tested positive})\)?

Among the 1,000,000 people, only 1 out of every 10,000 is sick. That gives us 100 sick people and 999,900 healthy people. Now let us test every single person and split them into groups: those diagnosed sick and those diagnosed healthy.

Healthy People (999,900 total)

Since the test is 99% effective, it makes a mistake on 1% of healthy people:

Diagnosed sick (false positive): \(999{,}900 \times 0.01 = 9{,}999\)
Diagnosed healthy (correct): \(999{,}900 \times 0.99 = 989{,}901\)

Sick people (100 total)

The test correctly identifies 99% of sick people:

Diagnosed sick (correct): \(100 \times 0.99 = 99\)
Diagnosed healthy (false negative): \(100 \times 0.01 = 1\)

Computing the Probability

Now look at the group of people who were diagnosed sick. This is the group you belong to (you tested positive). Using the icons from the diagram:

) =

(

) + (

)

10,098

≈ 0.0098 < 1%

In this group:

Diagnosed Sick	Count
Actually sick (true positive)	99
Actually healthy (false positive)	9,999
Total diagnosed sick	10,098

The probability that you are actually sick, given that you tested positive, is:

\[ P(\text{sick} \mid \text{tested positive}) = \frac{99}{99 + 9{,}999} = \frac{99}{10{,}098} \approx 0.0098 \]

That is less than 1%.

Surprising Result

You tested positive with a 99% effective test, but the probability that you are actually sick is less than 1%. The reason is that the disease is so rare (1 in 10,000) that the small number of false positives among the huge healthy population far outnumbers the true positives from the tiny sick population.

Probability Tree View

We can organize this calculation as a probability tree:

The four groups are:

Group	Count
Sick and diagnosed sick	99
Sick and diagnosed healthy	1
Healthy and diagnosed sick	9,999
Healthy and diagnosed healthy	989,901

Since you tested positive, you are in one of the first two rows of the “diagnosed sick” column. Most people in that column are actually healthy (9,999 vs 99). So the probability of being sick is only \(\frac{99}{10{,}098} < 1\%\).

Why Does This Happen?

The key insight is the base rate (also called the prior probability). The disease is extremely rare, affecting only 0.01% of the population. Even though the test is 99% accurate, 1% of a very large healthy population produces many more false positives than the true positives from the tiny sick population.

This is Bayes theorem in action: the probability of having the disease given a positive test depends not just on how good the test is, but also on how common the disease is in the first place.

Bayes Theorem Formula

Now let us write the general formula. Let \(A\) = sick and \(B\) = diagnosed sick. We want \(P(A \mid B)\).

From the conditional probability formula (covered in the previous lesson):

\[ P(A \mid B) = \frac{P(A \cap B)}{P(B)} \]

We need to find both the numerator and the denominator.

Numerator

Using the product rule: \(P(A \cap B) = P(A) \cdot P(B \mid A)\)

In our example this becomes:

\[ P(A \cap B) = P(\text{sick}) \cdot P(\text{diagnosed sick} \mid \text{sick}) \]

So at this point we have:

\[ P(A \mid B) = \frac{P(\text{sick}) \cdot P(\text{diagnosed sick} \mid \text{sick})}{P(\text{diagnosed sick}) = \text{ ?}} \]

We know the top. But what about the bottom? What is \(P(\text{diagnosed sick})\)?

Denominator

The people who are diagnosed sick come from two disjoint groups:

Those who are sick and correctly diagnosed sick
Those who are healthy and incorrectly diagnosed sick

Since you cannot be sick and healthy at the same time, these groups do not overlap. So we add their probabilities:

\[ P(\text{diagnosed sick}) = P(\text{sick} \cap \text{diagnosed sick}) + P(\text{not sick} \cap \text{diagnosed sick}) \]

Putting It Together

Now we substitute the denominator back into the formula:

\[ P(A \mid B) = \frac{P(\text{sick}) \cdot P(\text{diagnosed sick} \mid \text{sick})}{P(\text{sick} \cap \text{diagnosed sick}) + P(\text{not sick} \cap \text{diagnosed sick})} \]

We can expand each intersection in the denominator using the product rule as well:

\(P(\text{sick} \cap \text{diagnosed sick}) = P(\text{sick}) \cdot P(\text{diagnosed sick} \mid \text{sick})\)
\(P(\text{not sick} \cap \text{diagnosed sick}) = P(\text{not sick}) \cdot P(\text{diagnosed sick} \mid \text{not sick})\)

This gives us the full Bayes theorem:

\[ P(A \mid B) = \frac{P(A) \cdot P(B \mid A)}{P(A) \cdot P(B \mid A) + P(A') \cdot P(B \mid A')} \]

Plugging in the Numbers

\(P(A) = P(\text{sick}) = 0.0001\)
\(P(A') = P(\text{healthy}) = 0.9999\)
\(P(B \mid A) = P(\text{diagnosed sick} \mid \text{sick}) = 0.99\)
\(P(B \mid A') = P(\text{diagnosed sick} \mid \text{healthy}) = 0.01\)

\[ P(A \mid B) = \frac{0.0001 \times 0.99}{0.0001 \times 0.99 + 0.9999 \times 0.01} = \frac{0.000099}{0.000099 + 0.009999} \approx 0.0098 \]

This gives us exactly the same answer as before: less than 1%.

Review Questions

1. If the disease affected 1 in 100 people instead of 1 in 10,000, would \(P(\text{sick} \mid \text{tested positive})\) be higher or lower? Explain why.

Answer

It would be much higher. With 1 in 100 sick, out of 1,000,000 people there would be 10,000 sick and 990,000 healthy. True positives: \(10{,}000 \times 0.99 = 9{,}900\). False positives: \(990{,}000 \times 0.01 = 9{,}900\). So \(P(\text{sick} \mid \text{positive}) = \frac{9{,}900}{9{,}900 + 9{,}900} = 0.5\) (50%). A higher base rate means more true positives relative to false positives.

1. What would happen if the test were 99.9% effective instead of 99%? Compute \(P(\text{sick} \mid \text{tested positive})\) with the original 1 in 10,000 base rate.

Answer

With a 99.9% effective test: true positives = \(100 \times 0.999 = 99.9\), false positives = \(999{,}900 \times 0.001 = 999.9\). So \(P(\text{sick} \mid \text{positive}) = \frac{99.9}{99.9 + 999.9} \approx 0.091\) (about 9%). Better than before, but still surprisingly low because the disease is so rare.

1. Why do we compute “probability of sick given tested positive” rather than just looking at the test accuracy (99%)?

Answer

The test accuracy (99%) tells us \(P(\text{positive} \mid \text{sick})\), which is the probability of testing positive if you are sick. But what we actually want to know is the reverse: \(P(\text{sick} \mid \text{positive})\). These two probabilities are not the same. Bayes theorem lets us flip the conditional probability by accounting for the base rate of the disease.

1. A disease impacts 1% of the population. Researchers devised a test so that people with the disease test positive 95% of the time. People who do not have the disease test negative 90% of the time. If an individual receives a positive test result, what is the probability that they truly have the disease?

15.58%
42.76%
8.76%
90%

Answer

c. 8.76%. Using Bayes theorem: \(P(\text{sick} \mid \text{test}_{pos}) = \frac{P(\text{test}_{pos} \mid \text{sick}) \cdot P(\text{sick})}{P(\text{test}_{pos})}\). We have \(P(\text{sick}) = 0.01\), \(P(\text{test}_{pos} \mid \text{sick}) = 0.95\), \(P(\text{test}_{pos} \mid \text{not sick}) = 1 - 0.90 = 0.10\). Using the law of total probability: \(P(\text{test}_{pos}) = 0.95 \times 0.01 + 0.10 \times 0.99 = 0.0095 + 0.099 = 0.1085\). Therefore \(P(\text{sick} \mid \text{test}_{pos}) = \frac{0.0095}{0.1085} \approx 0.0876 = 8.76\%\). Even with a positive test, the low base rate means most positives are false positives.

Spam Detection Example

Here is another application of Bayes theorem: detecting spam emails.

Suppose we have a dataset of 100 emails. Out of these, 20 are spam and 80 are not spam (sometimes called “ham”). Without any additional information, the simplest classifier would say: “every email has a 20% chance of being spam.” That is all we know so far. We want something better.

Using a Feature: the Word “Lottery”

We notice that spam emails tend to contain certain words. Let us look at how many emails contain the word “lottery”:

Among the 20 spam emails: 14 contain “lottery”
Among the 80 ham emails: 10 contain “lottery”

Now the question is: if an email contains the word “lottery,” what is the probability it is spam?

Counting Approach

We only care about emails that contain “lottery.” There are \(14 + 10 = 24\) such emails. Among those 24, how many are spam? 14.

\[ P(\text{spam} \mid \text{lottery}) = \frac{14}{24} = \frac{7}{12} \approx 0.583 \]

So an email containing “lottery” has about a 58.3% chance of being spam. That is much higher than the baseline 20%.

Using the Bayes Theorem Formula

Let us verify this using the formula. Let \(A\) = spam and \(B\) = contains “lottery.”

First, we identify all the pieces:

\(P(A) = P(\text{spam}) = \frac{20}{100} = 0.2\) (the prior, our original knowledge before seeing the email)
\(P(A') = P(\text{not spam}) = 0.8\)
\(P(B \mid A) = P(\text{lottery} \mid \text{spam}) = \frac{14}{20} = 0.7\)
\(P(B \mid A') = P(\text{lottery} \mid \text{not spam}) = \frac{10}{80} = 0.125\)

Now we plug into Bayes theorem:

\[ P(\text{spam} \mid \text{lottery}) = \frac{P(\text{spam}) \cdot P(\text{lottery} \mid \text{spam})}{P(\text{spam}) \cdot P(\text{lottery} \mid \text{spam}) + P(\text{not spam}) \cdot P(\text{lottery} \mid \text{not spam})} \]

\[ = \frac{0.2 \times 0.7}{0.2 \times 0.7 + 0.8 \times 0.125} = \frac{0.14}{0.14 + 0.10} = \frac{0.14}{0.24} \approx 0.583 \]

This matches the counting approach exactly.

Prior and Posterior

We have seen Bayes theorem in action through two examples now. Let us formalize something that kept appearing in both.

There is something called the prior. That is the original probability you can calculate without knowing anything else. Then something happens. That something is the event. The event gives you information about the probability. With that information, you can calculate something called the posterior.

In math notation: the prior is \(P(A)\). An event can be called \(E\). The posterior is \(P(A \mid E)\).

The posterior is always a better estimation than the prior, because the event gave us information that we did not have before.

Spam Email

In the spam example, the prior was 20%. That was the number of spam emails divided by the total number of emails. The only thing we knew about an email, if we knew nothing else, was that it had a 20% probability of being spam. That is a very basic piece of information, and we needed to know a little more.

So we came up with a condition, an event: the email contains the word “lottery.”

Now the posterior changes. The probability that the email is spam, given that it contains the word “lottery,” is no longer the number of spam emails divided by the total number of emails. It is now the number of spam emails with the word “lottery” divided by the total number of emails with the word “lottery.” That is 14 out of 24, or about 58.3%. A much more accurate probability.

Medical Test

In the medical example, there was also a prior, an event, and a posterior. The prior was the original probability of being sick: \(0.0001\). The event was testing positive. The posterior was the probability that you were actually sick given that you tested positive: less than 1%.

Dice

You are rolling two dice and you want to find the probability that the sum is 10. We know that is \(\frac{3}{36}\). That is the prior.

Now the event says that the first die is a 6. All of a sudden the posterior changes. Now it is \(\frac{1}{6}\), because you live in a different, smaller sample space.

Coins

You want to find the probability that both coins land on heads. That is \(\frac{1}{4}\). That is the prior.

The event says the first coin landed on heads. Now you no longer have the bottom row of the table, and the posterior is \(\frac{1}{2}\).

Naive Bayes Model

Let us go back to the spam example. We had the prior (probability that an email is spam), the event (the email contains the word “lottery”), and the posterior (a new, better probability that the email is spam given that it contains “lottery”).

Obviously, we are not going to build a classifier with only one word. We want to use many words. So let us look at another one. Say we also use the word “winning” and calculate the posterior: \(P(\text{spam} \mid \text{winning})\).

Now how would we combine these two? How would we make a stronger classifier that uses both “lottery” and “winning”?

Problem with Multiple Words

The simple approach would be to calculate \(P(\text{spam} \mid \text{lottery and winning})\) by dividing the number of spam emails that contain both words by the total number of emails that contain both words.

Using Bayes theorem, we would need \(P(\text{lottery and winning} \mid \text{spam})\), which is the number of spam emails containing both words divided by the total number of spam emails.

But can you see the problem? Imagine we want to do this with not 2 words but 100 words: \(w_1, w_2, \ldots, w_{100}\). We would need to find emails that contain all 100 words at once. That is very hard. Maybe there are no such emails in our database, and we get 0 divided by 0. That is not a useful calculation.

Naive Assumption

The solution is to make a naive assumption: we assume that the appearances of the words are independent from each other.

This is obviously not true in reality. Words depend on each other. If the word “good” appears, “morning” is more likely to appear because “good morning” tends to appear together. So words are dependent.

But if we assume they are independent, the math works out much better. And we actually get pretty good results with this naive assumption. That is why it is called Naive Bayes.

How It Works

If the words are independent, then the probability of seeing both “lottery” and “winning” given spam is the product of the individual probabilities:

\[ P(\text{lottery and winning} \mid \text{spam}) = P(\text{lottery} \mid \text{spam}) \times P(\text{winning} \mid \text{spam}) \]

The same applies for ham. And this extends to any number of words:

\[ P(w_1, w_2, \ldots, w_n \mid \text{spam}) \approx P(w_1 \mid \text{spam}) \times P(w_2 \mid \text{spam}) \times \cdots \times P(w_n \mid \text{spam}) \]

That is the Naive Bayes algorithm.

Example with Two Words

Let us go back to our 100 emails (20 spam, 80 ham). We already know:

\(P(\text{spam}) = 0.2\), \(P(\text{ham}) = 0.8\)
\(P(\text{lottery} \mid \text{spam}) = \frac{14}{20} = 0.7\)
\(P(\text{lottery} \mid \text{ham}) = \frac{10}{80} = 0.125\)

Now for the word “winning”: 15 spam emails contain it and 8 ham emails contain it.

\(P(\text{winning} \mid \text{spam}) = \frac{15}{20} = 0.75\)
\(P(\text{winning} \mid \text{ham}) = \frac{8}{80} = 0.1\)

Plugging into Naive Bayes:

\[ P(\text{spam} \mid \text{lottery, winning}) = \frac{0.2 \times 0.7 \times 0.75}{0.2 \times 0.7 \times 0.75 + 0.8 \times 0.125 \times 0.1} \]

\[ = \frac{0.105}{0.105 + 0.01} = \frac{0.105}{0.115} \approx 0.913 \]

If an email contains both “lottery” and “winning,” there is a 91.3% chance it is spam. That is much higher than the 58.3% we got with just one word. Adding more evidence makes the classifier stronger.

Probability in Machine Learning

You may be wondering: why are we talking so much about probabilities? What do they have to do with machine learning?

It turns out that machine learning is a lot about probabilities. Many times in machine learning, what you want to do is calculate a probability of something given some other factors. That is a conditional probability.

In spam detection, you try to calculate the probability that an email is spam based on the words in the email, the recipients, the attachments, or other features. That is \(P(\text{spam} \mid \text{features})\).

In sentiment analysis, you want to determine if a piece of text is happy or sad. You find the probability that the text is happy given the words it contains: \(P(\text{happy} \mid \text{words})\).

In image recognition, you try to find out if an image contains a cat or not. You calculate the probability that there is a cat in the image based on its pixels: \(P(\text{cat} \mid \text{pixels})\).

These are all conditional probabilities.

What We Did with Spam is Machine Learning

When you look at the spam example at a high level, what really happened is that we created a machine learning classifier. The classifier gives us a calculation of the probability of one thing given another thing. And that is what machine learning is in many cases.

Think about image recognition. An image classifier tells you the probability that an image contains a cat based on the pixels. If you feed it a picture of a cat, it might say \(P(\text{cat} \mid \text{pixels}) = 0.9\). If you feed it a picture of a car, it might say \(P(\text{cat} \mid \text{pixels}) = 0.1\). So you determine that the second image is not a cat.

In the medical field, you might have demographics, symptoms, and test results for a patient, and you want to know if the patient is healthy. You calculate \(P(\text{healthy} \mid \text{symptoms, history})\). That is the same structure.

Generative Models

There is another big area of machine learning called generative machine learning. This is part of unsupervised learning, where you want to maximize probabilities rather than compute conditional ones.

In image generation, you want to maximize the probability that a bunch of pixels form a human face. In text generation, you want to maximize the probability that a bunch of words form sensible text about a certain topic.

These are all examples of how probability sits at the foundation of machine learning.

Key Insight

Bayes theorem is just a formula for updating probabilities when you learn something new. You start with a prior (what you believed before), observe evidence, and compute a posterior (what you should believe now). This is the foundation of spam filters, medical diagnosis, and many machine learning algorithms.

Lab: Monty Hall Problem

The Monty Hall problem is a classic puzzle that demonstrates how conditional probability can produce counter-intuitive results.

Setup

You are on a game show with three closed doors. Behind one door is a car (the prize). Behind the other two are goats. You pick a door, say Door 1. The host, who knows what is behind each door, opens a different door (say Door 3) revealing a goat. The host then asks: “Would you like to switch to Door 2?”

The question is: should you switch or stay?

Most People Think It Does Not Matter

Your first instinct might be: “There are two doors left, so it is 50/50 either way.” But this is wrong. The host’s action gives you new information, and that changes the probabilities.

Why Switching Wins 2/3 of the Time

When you first pick a door, you have a \(\frac{1}{3}\) chance of being right and a \(\frac{2}{3}\) chance of being wrong. The host then removes one wrong door from the remaining two. That \(\frac{2}{3}\) probability of the car being behind one of the other doors is now concentrated on the single remaining door.

If you stay, you win only if your original pick was correct: \(P(\text{win}) = \frac{1}{3}\)
If you switch, you win whenever your original pick was wrong: \(P(\text{win}) = \frac{2}{3}\)

So switching doubles your chance of winning.

Still Not Convinced?

Imagine the same game but with 20 doors instead of 3. You pick one door. The host, who knows where the car is, opens 18 other doors, all revealing goats. Now you are left with two doors: the one you originally chose and the one the host deliberately kept closed.

Your original pick had a \(\frac{1}{20}\) chance of being correct. The other 19 doors collectively had \(\frac{19}{20}\). The host just eliminated 18 of those 19, concentrating all that probability onto the single remaining door. Would you switch now? Of course you would. The 3-door version works exactly the same way, just with smaller numbers.

Connection to Bayes Theorem

This is conditional probability in action. After the host opens a door, the probability distribution over the remaining doors is not uniform. The host’s action (which is conditioned on knowing where the car is) updates the probabilities, just like in the medical test example where a positive test result updates the probability of being sick.

Try It Yourself

If this still feels wrong, try simulating it. Play the game 30 times with each strategy (always stay vs always switch) and count your wins. You will see that switching wins about twice as often.

Simulating with Python

You can run this code locally to simulate the Monty Hall problem:

import numpy as np

def monty_hall_simulation(n_games=100):
    stay_wins = 0
    switch_wins = 0

    for i in range(n_games):
        car = np.random.randint(0, 3)
        player_pick = np.random.randint(0, 3)

        if player_pick == car:
            stay_wins += 1
        else:
            switch_wins += 1

    print(f"After {n_games} games:")
    print(f"  Stay win rate:   {stay_wins/n_games:.2%} (expected ≈ 33.3%)")
    print(f"  Switch win rate: {switch_wins/n_games:.2%} (expected ≈ 66.7%)")

monty_hall_simulation(1000)

Interactive Simulation

Choose the number of games to simulate (up to 1,000), then observe how both strategies perform:

MPlot = import("https://esm.sh/@observablehq/plot@0.6")

montySlider = {
  const container = html`<div style="display:flex; align-items:center; gap:12px; margin:1em 0; font-family:'PT Sans',sans-serif;">
    <label style="font-weight:600; white-space:nowrap;">Number of games:</label>
    <input type="range" id="monty-slider" min="10" max="1000" value="100" step="10" style="flex:1; min-width:200px;">
    <span id="monty-slider-val" style="font-weight:700; min-width:40px;">100</span>
  </div>`;

  const slider = container.querySelector('#monty-slider');
  const valEl = container.querySelector('#monty-slider-val');

  slider.addEventListener('input', () => {
    valEl.textContent = slider.value;
    mutable montyN = parseInt(slider.value);
  });

  return container;
}

mutable montyN = 100

montySimData = {
  const n = montyN;
  let stayWins = 0, switchWins = 0;
  const stayData = [];
  const switchData = [];

  for (let i = 1; i <= n; i++) {
    const car = Math.floor(Math.random() * 3);
    const pick = Math.floor(Math.random() * 3);
    if (pick === car) stayWins++;
    else switchWins++;
    stayData.push({game: i, rate: stayWins / i, strategy: "Stay"});
    switchData.push({game: i, rate: switchWins / i, strategy: "Switch"});
  }

  return {stayData, switchData, stayRate: stayWins / n, switchRate: switchWins / n, n};
}

html`<div style="border:2px solid #ddd; border-radius:8px; padding:16px; margin:1em 0; background:#fafafa; font-family:'PT Sans',sans-serif;">
  <div style="display:flex; gap:24px; margin-bottom:16px;">
    <div style="flex:1; padding:12px; background:#fff; border:1px solid #eee; border-radius:6px; text-align:center;">
      <div style="font-weight:600; color:#CC0000;">Stay Strategy</div>
      <div style="font-size:1.5em; font-weight:700; margin-top:4px;">${(montySimData.stayRate * 100).toFixed(1)}%</div>
    </div>
    <div style="flex:1; padding:12px; background:#fff; border:1px solid #eee; border-radius:6px; text-align:center;">
      <div style="font-weight:600; color:#22176F;">Switch Strategy</div>
      <div style="font-size:1.5em; font-weight:700; margin-top:4px;">${(montySimData.switchRate * 100).toFixed(1)}%</div>
    </div>
  </div>
  <div style="text-align:center; color:#666; font-size:0.9em;">After ${montySimData.n} games (drag slider to re-run)</div>
</div>`

html`<div style="display:flex; gap:12px; margin:12px 0;">
  <div style="flex:1; border:1px solid #ddd; border-radius:6px; padding:12px; background:#fff; min-width:0;">
    <p style="font-weight:600; font-size:0.9em; margin:0 0 8px 0; color:#CC0000;">Stay Strategy</p>
    ${MPlot.plot({
      height: 240,
      marginBottom: 40,
      marginLeft: 50,
      style: {width: "100%"},
      x: {label: "Game number", domain: [1, montySimData.n]},
      y: {label: "Win rate", domain: [0, 1], tickFormat: ".0%"},
      marks: [
        MPlot.line(montySimData.stayData, {x: "game", y: "rate", stroke: "#CC0000", strokeWidth: 2}),
        MPlot.ruleY([1/3], {stroke: "#999", strokeDasharray: "5,5", strokeOpacity: 0.6}),
        MPlot.text([{x: montySimData.n * 0.85, y: 0.37}], {x: "x", y: "y", text: ["1/3"], fill: "#999", fontSize: 10})
      ]
    })}
  </div>
  <div style="flex:1; border:1px solid #ddd; border-radius:6px; padding:12px; background:#fff; min-width:0;">
    <p style="font-weight:600; font-size:0.9em; margin:0 0 8px 0; color:#22176F;">Switch Strategy</p>
    ${MPlot.plot({
      height: 240,
      marginBottom: 40,
      marginLeft: 50,
      style: {width: "100%"},
      x: {label: "Game number", domain: [1, montySimData.n]},
      y: {label: "Win rate", domain: [0, 1], tickFormat: ".0%"},
      marks: [
        MPlot.line(montySimData.switchData, {x: "game", y: "rate", stroke: "#22176F", strokeWidth: 2}),
        MPlot.ruleY([2/3], {stroke: "#999", strokeDasharray: "5,5", strokeOpacity: 0.6}),
        MPlot.text([{x: montySimData.n * 0.85, y: 0.70}], {x: "x", y: "y", text: ["2/3"], fill: "#999", fontSize: 10})
      ]
    })}
  </div>
</div>`

Drag the slider to increase the number of games. As you go from 10 to 1,000, the red line (stay) settles around 33% and the blue line (switch) settles around 67%. The more games you simulate, the closer both lines get to their expected values.

Lab: Naive Bayes Spam Detection

In this lab you will implement the Naive Bayes algorithm for a spam detection problem, directly applying Bayes theorem. You will build a classifier that determines whether an email is spam or ham (not spam) based on the words it contains.

Download the dataset: emails.csv

Problem Setup

The goal is to classify emails as spam (\(1\)) or ham (\(0\)). Using Bayes theorem:

\[ P(\text{spam} \mid \text{email}) = \frac{P(\text{email} \mid \text{spam}) \cdot P(\text{spam})}{P(\text{email})} \]

Since \(P(\text{email})\) appears in both the spam and ham calculations, you only need to compare:

\(P(\text{spam}) \cdot P(\text{email} \mid \text{spam})\)
\(P(\text{ham}) \cdot P(\text{email} \mid \text{ham})\)

The naive assumption is that each word in an email appears independently of the others:

\[ P(\text{email} \mid \text{spam}) = P(\text{word}_1 \mid \text{spam}) \cdot P(\text{word}_2 \mid \text{spam}) \cdots P(\text{word}_n \mid \text{spam}) \]

This assumption is false (words are correlated), but it leads to a simple and surprisingly effective algorithm.

Loading and Exploring Data

import numpy as np
import pandas as pd
from nltk.corpus import stopwords
from nltk import word_tokenize
import string

dataframe_emails = pd.read_csv('../../media/emails.csv')
dataframe_emails.head()

	text	spam
0	Subject: naturally irresistible your corporate...	1
1	Subject: the stock trading gunslinger fanny i...	1
2	Subject: unbelievable new homes made easy im ...	1
3	Subject: 4 color printing special request add...	1
4	Subject: do not have money , get software cds ...	1

print(f"Number of emails: {len(dataframe_emails)}")
print(f"Proportion of spam emails: {dataframe_emails.spam.sum()/len(dataframe_emails):.4f}")
print(f"Proportion of ham emails: {1-dataframe_emails.spam.sum()/len(dataframe_emails):.4f}")

Number of emails: 5728
Proportion of spam emails: 0.2388
Proportion of ham emails: 0.7612

The dataset is unbalanced: about 24% spam and 76% ham.

Preprocessing

Every email starts with “Subject:” which we remove. We shuffle the data for proper train/test splitting, then tokenize the text by removing stopwords and punctuation.

def preprocess_emails(df):
    """Shuffle the dataset and extract email content and labels."""
    df = df.sample(frac=1, ignore_index=True, random_state=42)
    X = df.text.apply(lambda x: x[9:]).to_numpy()
    Y = df.spam.to_numpy()
    return X, Y

X, Y = preprocess_emails(dataframe_emails)

def preprocess_text(X):
    """Remove stopwords and punctuation, tokenize text."""
    stop = set(stopwords.words('english') + list(string.punctuation))
    if isinstance(X, str):
        X = np.array([X])
    X_preprocessed = []
    for email in X:
        email = np.array([i.lower() for i in word_tokenize(email) if i.lower() not in stop]).astype(X.dtype)
        X_preprocessed.append(email)
    if len(X_preprocessed) == 1:
        return X_preprocessed[0]
    return X_preprocessed

# This may take a minute to run
X_treated = preprocess_text(X)

email_index = 989
print(f"Before: {X[email_index][:80]}...")
print(f"After: {X_treated[email_index][:10]}")

Before: marketing for your espeak session  vince :  thanks for your time earlier this we...
After: ['marketing' 'espeak' 'session' 'vince' 'thanks' 'time' 'earlier' 'week'
 'looking' 'forward']

Train/Test Split

TRAIN_SIZE = int(0.80 * len(X_treated))

X_train = X_treated[:TRAIN_SIZE]
Y_train = Y[:TRAIN_SIZE]
X_test = X_treated[TRAIN_SIZE:]
Y_test = Y[TRAIN_SIZE:]

print(f"Proportion of spam in train: {sum(Y_train == 1)/len(Y_train):.4f}")
print(f"Proportion of spam in test: {sum(Y_test == 1)/len(Y_test):.4f}")

Proportion of spam in train: 0.2431
Proportion of spam in test: 0.2216

Implementing Naive Bayes

Step 1: Build a word frequency dictionary. For each word, count how many spam and ham emails contain it. Start each count at 1 to avoid zeros in the product.

def get_word_frequency(X, Y):
    """Build a dictionary counting word appearances in spam and ham emails."""
    word_dict = {}
    for i in range(len(X)):
        email = set(X[i])
        cls = Y[i]
        for word in email:
            if word not in word_dict:
                word_dict[word] = {"spam": 1, "ham": 1}
            if cls == 0:
                word_dict[word]["ham"] += 1
            if cls == 1:
                word_dict[word]["spam"] += 1
    return word_dict

word_frequency = get_word_frequency(X_train, Y_train)
class_frequency = {'ham': sum(Y_train == 0), 'spam': sum(Y_train == 1)}
print(f"Vocabulary size: {len(word_frequency)} words")
print(f"Training set: {class_frequency}")

Vocabulary size: 33812 words
Training set: {'ham': np.int64(3468), 'spam': np.int64(1114)}

Step 2: Compute \(P(\text{word} \mid \text{class})\):

def prob_word_given_class(word, cls, word_frequency, class_frequency):
    """Calculate P(word | class)."""
    return word_frequency[word][cls] / class_frequency[cls]

print(f"P(lottery | spam) = {prob_word_given_class('lottery', 'spam', word_frequency, class_frequency):.6f}")
print(f"P(lottery | ham) = {prob_word_given_class('lottery', 'ham', word_frequency, class_frequency):.6f}")
print(f"P(schedule | spam) = {prob_word_given_class('schedule', 'spam', word_frequency, class_frequency):.6f}")
print(f"P(schedule | ham) = {prob_word_given_class('schedule', 'ham', word_frequency, class_frequency):.6f}")

P(lottery | spam) = 0.008079
P(lottery | ham) = 0.000288
P(schedule | spam) = 0.008977
P(schedule | ham) = 0.102941

Step 3: Compute \(P(\text{email} \mid \text{class})\) by multiplying word probabilities:

def prob_email_given_class(treated_email, cls, word_frequency, class_frequency):
    """Calculate P(email | class) using naive independence assumption."""
    prob = 1
    for word in treated_email:
        if word in word_frequency:
            prob *= prob_word_given_class(word, cls, word_frequency, class_frequency)
    return prob

example_email = "Click here to win a lottery ticket and claim your prize!"
treated = preprocess_text(example_email)
print(f"Email: {example_email}")
print(f"Tokens: {treated}")
print(f"P(email | spam) = {prob_email_given_class(treated, 'spam', word_frequency, class_frequency):.4e}")
print(f"P(email | ham) = {prob_email_given_class(treated, 'ham', word_frequency, class_frequency):.4e}")

Email: Click here to win a lottery ticket and claim your prize!
Tokens: ['click' 'win' 'lottery' 'ticket' 'claim' 'prize']
P(email | spam) = 5.3885e-11
P(email | ham) = 1.2428e-15

Step 4: The full classifier. Compare \(P(\text{spam}) \cdot P(\text{email} \mid \text{spam})\) vs \(P(\text{ham}) \cdot P(\text{email} \mid \text{ham})\):

def naive_bayes(treated_email, word_frequency, class_frequency):
    """Classify an email as spam (1) or ham (0)."""
    prob_email_spam = prob_email_given_class(treated_email, 'spam', word_frequency, class_frequency)
    prob_email_ham = prob_email_given_class(treated_email, 'ham', word_frequency, class_frequency)

    p_spam = class_frequency['spam'] / (class_frequency['spam'] + class_frequency['ham'])
    p_ham = class_frequency['ham'] / (class_frequency['spam'] + class_frequency['ham'])

    spam_likelihood = p_spam * prob_email_spam
    ham_likelihood = p_ham * prob_email_ham

    return 1 if spam_likelihood >= ham_likelihood else 0

for email in ["Click here to win a lottery ticket and claim your prize!",
              "Our meeting will happen in the main office. Please be there in time."]:
    treated = preprocess_text(email)
    pred = "spam" if naive_bayes(treated, word_frequency, class_frequency) == 1 else "ham"
    print(f"'{email}' -> {pred}")

'Click here to win a lottery ticket and claim your prize!' -> spam
'Our meeting will happen in the main office. Please be there in time.' -> ham

Model Performance

Y_pred = [naive_bayes(email, word_frequency, class_frequency) for email in X_test]

true_positives = sum(1 for yt, yp in zip(Y_test, Y_pred) if yt == 1 and yp == 1)
true_negatives = sum(1 for yt, yp in zip(Y_test, Y_pred) if yt == 0 and yp == 0)
accuracy = (true_positives + true_negatives) / len(Y_test)

print(f"True positives (spam correctly detected): {true_positives}")
print(f"True negatives (ham correctly detected): {true_negatives}")
print(f"Accuracy: {accuracy:.4f}")

True positives (spam correctly detected): 249
True negatives (ham correctly detected): 723
Accuracy: 0.8482

Even with the naive independence assumption, the model achieves about 85% accuracy. Not bad for such a simple algorithm built entirely from Bayes theorem.

Try It Yourself

email = "Congratulations! You have been selected to receive a free vacation package. Click here now!"
treated = preprocess_text(email)
pred = "spam" if naive_bayes(treated, word_frequency, class_frequency) == 1 else "ham"
print(f"Email: {email}")
print(f"Prediction: {pred}")

Email: Congratulations! You have been selected to receive a free vacation package. Click here now!
Prediction: spam