Fallacy of the Week: The Base Rate Fallacy

Dear Doctor Math,
If a doctor tells you that you tested positive for something and that the test is 99% accurate, does that mean you have a 99% chance of having the disease? Just curious.
MTG

I should probably begin by reiterating that I'm not actually a doctor, at least not that kind (see earlier post), so please don't take what I'm about to say as medical advice. But basically the answer is no, or at least you can't tell without further information. You see, what your doctor may have omitted telling you is the base rate for the disease in question, that is, the general probability of having the disease without the extra information that you tested positive for it. If that base rate is really low, even a very accurate test isn't a strong indicator of having the disease. Let me illustrate with some numbers:

Let's say that, on average, 1 out of every 1 million people suffers from psychogenic dwarfism. So, for this condition your base rate is , or .0001%. Now, you go in for a physical and the doctor says that you tested positive for this debilitating condition, and that the test is 99% accurate. There's some room for interpretation as to what that means exactly, but let's take it to mean that (1) if you actually have the condition, you'll definitely test positive, and (2) if you don't have the condition, there's a 99% chance you'll test negative. So, what are your odds? Well, imagine that 100 million people go in for tests. We know that about 100 of them will actually be psychogenic dwarves. Of the remaining 99,999,900 people, however, about 1% will test positive even though they don't have it. So that means 999,999 false positives compared to only 100 true positives. The total number of people testing positive is 100 + 999,999, or 1,000,099, and only 100 of them actually have the disease. Since all you know is that you tested positive, you don't know which of these people you are, so your chance of actually being positive is , about 1 in 10,000, or .01%. Bottom line: it's still very unlikely that you have the condition, even though you tested positive for it and the test is very accurate, so don't go out and sell all your normal-sized clothes just yet.

The basic rule here is always to consider the number of true positives relative to the total number of people who would test positive, true or false. What we've seen in this example, and what frequently turns out to be the case, is that even a test that sounds like a sure thing can end up producing way more false positives than true ones, just because there aren't that many true positives out there to discover. (For another example, ask the Department of Homeland Security about their terrorist-detecting techniques.) I think part of the problem here is that we're dealing with numbers that we don't have much intuitive grasp for. I mean, "one in a million" basically means it won't ever happen, right? But "99 percent accurate" means it must be true. So how do you decide? The nice thing about math is that can't get bullied around by intimidating sounding numbers like these; it just puts them in their relative place. Remember, probability is ultimately all about information, and it should take a lot of evidence to convince us of something extremely unlikely.

-DrM