Poll: Accuracy of Test Interpretation

[Wrath of the Swarm]

Let's say that I went for an annual medical checkup, and the doctor wanted to know if I had a particular disease that affects one out of every thousand people. To check, he performed a blood test that is known to be about 99% accurate. The test results came back positive. The doctor concluded that I have the disease.

How likely is it that the diagnosis is correct?

It's best if you don't sit down to work it out. Just give your honest opinion about what you think is likely. If you happen to know the formula that gives the correct answer, feel free to use it.

 

[Wrath of the Swarm]

S

P

O

I

L

E

R

S

I'll post the correct answer in three days when the poll shuts down.

 

[slimshady2357]

Too easy, but what's the chances of you having the disease if you get three positives in a row?

Adam

 

[Wrath of the Swarm]

A lot of doctors can't do this problem correctly. I expect the forumites to do better, but I think I'll still make my point.

(After three positive results, the chances of having the disease are about 99.9%.)

[edit] (This assumes, of course, that each test is independent of the others, which is not a realistic assumption. Still, tests with such high degrees of accuracy are generally unrealistic as well.)

 

[Rolfe]

Hmmm, big piece of information missing. Are you showing any clinical signs of the disease, or is your probability of infection that of the general background population?

The thing is, to get the predictive value of a test (which is what Wrath is asking), you need to know the incidence of the condition in the population representative of the individual being tested. This is obviously higher if that population is "sick people with clinical signs typical of the disease in question". In fact, the relevant figure is the clinical probability that this individual is affected.

I also need to know if the figure of "99% accurate" refers particularly to specificity. Specificity is the percentage of positive results which are correct (i.e. identify an affected individual) and is the important parameter here. Sensitivity, on the other hand, is the percentage of negative results which are correct (i.e. identify an unaffected individual). The two together may be loosely combined and referred to as "accuracy", but if we are talking about a positive result, it is the specificity figure one needs to know.

However, if we assume for the moment that Wrath's probability of being affected is the same as the general population, that is 0.1%, and that the specificity of the test is 99% (I don't really care about sensitivity in this situation), a positive result is 90.98% likely to be wrong and 9.02% likely to be correct.

So there is only (approximately) a 9% chance that Wrath actually has the condition.

(I'm sorry, I cheated, I have a spreadsheet on my computer to produce this information and a pretty graph that demonstrates how the predictive value of a positive or a negative result varies with clinical probability of infection, and sensitivity and specificity of the test.)

Rolfe.

 

[geni]

It's no good becase of the way Wrath of the Swarm worded the question there is more than one correct answer

 

[Rolfe]

It's no good becase of the way Wrath of the Swarm worded the question there is more than one correct answer

Well, I did state some assumptions. Did I leave anything out?

Rolfe.

 

[Wrath of the Swarm]

No, there's only one correct answer.

Rolfe, you are both an idiot and a jerk. Now that you've posted an answer, the poll is completely useless.

(Oh, and your requests for more information are pointless. The information provided is more than sufficient to answer the question. Besides, this is the question that doctors have such trouble with - I don't think adding more would actually help them.)

 

[Rolfe]

Now that it's been explained, you don't get to feel so superior. What a shame.

(And I didn't ask for more information, I asked for better information.)

Rolfe.

 

[Wrath of the Swarm]

You mean now that it's been explained, we can no longer demonstrate that most people here have no grasp of probability theory as it applies to medical diagnosis?

Yep.

 

[Deetee]

So what is Wrath's "point", anyhow?

Since in his hypothetical scenario he deliberately had the doctor jump to the wrong conclusion, I suspect he is just out to try and rubbish the poor medics again.

I suspect that docs know a bit more about sensitivity, specificity, predicitive values, odds ratios etc than WoS gives them credit for, anyway.

 

[geni]

Well, I did state some assumptions. Did I leave anything out?

Rolfe.

Doing the maths the way I am suggests that you get wrath's answers by assuming the error is in false posertives. Playing around with false negativs screws the results.

And Wrath you had already given away way to much data in your posts

 

[geni]

wrong button

 

[Rolfe]

That was exactly the assumption I made. That the 99% "accuracy" (meaningless term, really) quoted was actually a specificity figure. If it included the sensitivity and the specificity was actually something different, then the result is void.

Anyway, five out of five posters already got it right while I was composing my post and before I hit the send button.

Rolfe.

 

[geni]

You mean now that it's been explained, we can no longer demonstrate that most people here have no grasp of probability theory as it applies to medical diagnosis?

Yep.

Er no beacuse most of the time information wont be presented the way you presented it (If it is You find the person responcible and tell them to get into the advertising bussness if they already are in the advertising bissness then why were you beliving them in the first place?).

All you've done is present a waiter pocketing the tip type problem

 

[Brian the Snail]

You mean now that it's been explained, we can no longer demonstrate that most people here have no grasp of probability theory as it applies to medical diagnosis?

Yep.

Actually, before Rolfe revealed the answer there were at least 4 votes cast, all of which gave the correct answer.

So what makes you think that most people here would have got the answer wrong?

 

[Wrath of the Swarm]

The accuracy covers both true positives and true negatives. If I had specified the rate of alpha error only, you would have needed to know the beta error. But since I didn't, you didn't.

 

[Wrath of the Swarm]

So what is Wrath's "point", anyhow?

Since in his hypothetical scenario he deliberately had the doctor jump to the wrong conclusion, I suspect he is just out to try and rubbish the poor medics again.

It must be nice to be able to psychically determine what someone's position is before they state it and tear apart the holes in the arguments they haven't made yet.

Have you considered trying out for the million? A debating technique like that could make you a very rich person.

 

[Soapy Sam]

And to add to the chaos, I just guesstimated 10% before I read the rest of the thread. So it's unanimous.
Wrath- I'd suggest you are playing to the wrong audience ith this one. Most folk here will have been reading J.A.Paulos and Martin Gardner for years. You said you expected us to beat GPs and you were right, but I think you underestimated by how much. You would really need a more random test group. (And a larger one of course).

 

[Wrath of the Swarm]

Actually, before Rolfe revealed the answer there were at least 4 votes cast, all of which gave the correct answer.

So what makes you think that most people here would have got the answer wrong?

Most medical students (and a significant fraction of doctors) get the question wrong.

Given that people here can check the other answers, take the time to work out the answer fully, and generally are better educated in statistics and critical thought, I figured many more people than normal would get the right answer. I expected the numbers of wrong answers to still be significant, though.

Since I clearly stated that I would post the correct answer when the poll cleared, it was extremely rude of Rolfe to post the explanation. After all, if there's the least chance it could be used to support a position she doesn't like, she couldn't permit it to continue unchecked, could she?