Statistical Numeracy

[marting]

It's no wonder people don't trust statistics. It seems far too many are statistically innumerate.

Measuring Risk Literacy: The Berlin Numeracy Test

http://journal.sjdm.org/11/11808/jdm11808.pdf

The Berlin Numeracy Test was found to be the strongest predictor of comprehension
of everyday risks (e.g., evaluating claims about products and treatments

Fascinating, but scary read that statistical innumeracy is widespread among professionals that presumably should understand this.

But I have a quibble. Take this question on page 46:

Imagine that you see the following advertisement for a
new toothpaste:
Zendil—50% reduction in occurrence of gum inflammation. Zendil is a new toothpaste to prevent gum inflammation. Half as many people who used Zendil developed
gum inflammation when compared to people using a different toothpaste.
Which one of the following would best help you evaluate how much a person could benefit from using Zendil?

1. The risk of gum inflammation for people who do not
use Zendil

2. The risk of gum inflammation for people who use a
different brand of toothpaste for the same purpose

selections 3-6 omitted as ludicrous.

Here's the problem. Answer 1 is best if one assumes everyone brushes their teeth with toothpaste. Answer 2 is best if one assumes a significant number of people do not use toothpaste since the question excludes those. However, it also refers to people that use toothpaste specifically to reduce gum inflammation which may skew results.

So the question seems quite problematic to me. BTW, the "correct" answer is #1

 

[Puppycow]

Both pieces of information would appear to have relevance to the question. Or could help you to answer the question. However, only answer 2 is directly applicable to the claim being made in the advertisement.

Also, "people who do not use Zendil" includes both people who use a different brand of toothpaste, and people who don't use any toothpaste. Do those groups have the same risk of inflammation?

Finally, in the real world there's also the practical skeptical question of whether to take a claim made by an advertiser at face value or not.

 

[marting]

Both pieces of information would appear to have relevance to the question. Or could help you to answer the question. However, only answer 2 is directly applicable to the claim being made in the advertisement.

But answer 1 is claimed to be correct.

The new and improved "Berlin Numeracy Test" doesn't have any of these ambiguities and are pretty straightforward. Here's an example:

In a forest 20% of mushrooms are red, 50% brown
and 30% white. A red mushroom is poisonous with
a probability of 20%. A mushroom that is not red
is poisonous with a probability of 5%. What is the
probability that a poisonous mushroom in the forest
is red?

Hard to believe how poorly professionals that need to communicate risk to others do on these. Is the education system that broken?

 

[Puppycow]

In a forest 20% of mushrooms are red, 50% brown
and 30% white. A red mushroom is poisonous with
a probability of 20%. A mushroom that is not red
is poisonous with a probability of 5%. What is the
probability that a poisonous mushroom in the forest
is red?

50%. Do I win? (5% of 80 = 20% of 20)

I do agree that a lot of people seem to be innumerate.

And you don't need to know fancy math like algebra or calculus to be numerate.

 

[marting]

50%. Do I win? (5% of 80 = 20% of 20)

I do agree that a lot of people seem to be innumerate.

And you don't need to know fancy math like algebra or calculus to be numerate.

Right answer. No you don't need any fancy math, just a minimal amount of reasoning and 8th grade arithmetic.

 

[Hlafordlaes]

Right answer. No you don't need any fancy math, just a minimal amount of reasoning and 8th grade arithmetic.

I didn't peek and got the 50%, but I heard rusty gears creaking. What is innumeracy-by-aging called? "Easy mark"?

 

[Dr. Keith]

I didn't peek and got the 50%, but I heard rusty gears creaking. What is innumeracy-by-aging called? "Easy mark"?

Let me know when you stop hearing the gears and I will help you set up a reverse mortgage on your home. They work really well for some people.

 

[Bikewer]

I read a book called “Innumerancy” some years back, and the primary thrust was about how statistics tend to be misused by the media, and how people’s poor understanding of same causes a lot of misinformation.

We used to have two newspapers here in St. Louis, the right-leaning Globe-Democrat, and the left-leaning Post-Dispatch.
The Globe ran a big headline one year, “Homicides in St. Louis County up 100%”.
Pretty attention-grabbing. And exactly correct.... the previous year we’d had one, and this particular year two.....

 

[Jim_MDP]

I didn't peek and got the 50%, but I heard rusty gears creaking. What is innumeracy-by-aging called? "Easy mark"?

Better than me...I misread the Q as total % for picking death (8%).
My gears are seized. [emoji3]

 

[marting]

Better than me...I misread the Q as total % for picking death (8%).
My gears are seized. [emoji3]

easy to misread. I'm pretty good with numbers but it's mostly because I've done technical work and computer programming most of my life. OTOH, my wife runs circles around me in lots of areas. When watching a movie she'll identify various actors and what they had also been in. I'm almost clueless and have long had poor facial/name recognition. I'm quite fascinated with the variety of cognitive skills/traits different people exhibit. I suspect the large variety has significant survival value for the species.

 

[bruto]

Call me innumerate, but I don't quite agree with the Zendil one. If you answer 1, you get some answer for sure, but since it includes an unspecified number of people who use no toothpaste at all, or unspecified toothpaste, it does not provide as useful an answer as the ad claim for those who do. Answer 2 inserts spurious information, since we cannot parse the purpose of tooth brushing that way.

I don't think either of those answers surpasses the advertised claim for usefulness (presuming the claim is honest). But none of the answers is truly satisfactory because in all cases, it seems to be stipulated that all toothpastes except for Zendil are the same, and considered in a group, or that the "other" toothpaste might not be Zendil's best competitor. Any test that simply specifies "an other toothpaste" could have results skewed, either by lumping all tootpastes together, and outweighing the good ones with the bad, or even by simply choosing the worst as "an other." A specialty toothpaste used by few could even be better, and invisible in a lump comparison.

That could be settled if the claim were "any other toothpaste," again of course assuming the claim to be true. Until that is done, answer 1 produces the result least likely to be false, but still possible to be misleading.

 

[Modified]

...What is the
probability that a poisonous mushroom in the forest
is red?...

The interpretation is obvious in this case, but this is still very poor phrasing. "Given a poisonous mushroom selected at random from the forest, what is the probability that it is red?" would be better.

 

[Puppycow]

We used to have two newspapers here in St. Louis, the right-leaning Globe-Democrat, and the left-leaning Post-Dispatch.
The Globe ran a big headline one year, “Homicides in St. Louis County up 100%”.
Pretty attention-grabbing. And exactly correct.... the previous year we’d had one, and this particular year two.....

That sounds very low for a county of that size in America!

How long ago was that, anyway?

In 2020:

St. Louis' 2020 Homicide Rate Is Highest in 50 Years

ST. LOUIS (AP) — St. Louis recorded its worst homicide rate in the past 50 years even though the total number of homicides during 2020 fell just short of the city's all-time record.

Police said 262 people were killed in St. Louis last year — five less than the record of 267 set in 1993. But because the city's population has declined since 1993, the homicide rate was much higher in 2020.

 

[Bikewer]

About 1968, when I started my police career. Burglary was the big crime in the county at the time, homicide almost unheard of.

 

[lomiller]

Both pieces of information would appear to have relevance to the question. Or could help you to answer the question. However, only answer 2 is directly applicable to the claim being made in the advertisement.

Also, "people who do not use Zendil" includes both people who use a different brand of toothpaste, and people who don't use any toothpaste. Do those groups have the same risk of inflammation?

Finally, in the real world there's also the practical skeptical question of whether to take a claim made by an advertiser at face value or not.

The question isn't as clear as it should be but the way I read it #2 is incorrect making #1 the correct answer.


The way I read the initial statement of the problem Zendil isn't being compared to a specific brand. It is it's entirely possible for it to be more effective than other toothpaste in general while still being less effective than a specific brand.

But answer 1 is claimed to be correct.

The new and improved "Berlin Numeracy Test" doesn't have any of these ambiguities and are pretty straightforward. Here's an example:

In a forest 20% of mushrooms are red, 50% brown
and 30% white. A red mushroom is poisonous with
a probability of 20%. A mushroom that is not red
is poisonous with a probability of 5%. What is the
probability that a poisonous mushroom in the forest
is red?

Hard to believe how poorly professionals that need to communicate risk to others do on these. Is the education system that broken?

Building on this to explain my point above.

20% of are red
40% are brown
30% are white
10% are green

20% of red mushrooms are poisonous
5% of all other mushrooms are poisonous

Q: Is a red mushroom safer than the green one.
A: The information provided isn't sufficient to tell us because 5% of all other mushrooms could mean

0% of brown mushrooms are poisonous
0% of white mushrooms are poisonous
62.5% of green mushrooms are poisonous

 

[Roboramma]

All probabilities are about lack of information.

Each individual mushroom is either poisonous or not.

You know that 5% of non-red mushrooms are poisonous. The fact that you don't know anything more than that doesn't mean you don't know that. So, no, you don't need the distribution to answer your question.

 

[lomiller]

All probabilities are about lack of information.

Each individual mushroom is either poisonous or not.

You know that 5% of non-red mushrooms are poisonous. The fact that you don't know anything more than that doesn't mean you don't know that. So, no, you don't need the distribution to answer your question.

You most certainly DO need that distribution to answer the question of whether red mushrooms are safer than green ones. Likewise in the original question you would need to know the occurrence of gum inflammation of both brands in question to decide if Zendil is better. Since you don't have this information #2 is the wrong answer. The information available doesn't allow you to reach any conclusion.

 

[marting]

Here's a hypothetical demonstrating statistical numeracy:

There are two, prospective DBRTCs, properly run for treatment A and treatment B. The treatment A group N=1000. The treatment B group N=100. Both studies showed efficacy of reducing mortality and achieved statistical significance.

Group A study had a p value of .03, Group B's p value was .04.

Given only the above information which treatment is probably most effective?

 

[Elaedith]

Here's a hypothetical demonstrating statistical numeracy:

There are two, prospective DBRTCs, properly run for treatment A and treatment B. The treatment A group N=1000. The treatment B group N=100. Both studies showed efficacy of reducing mortality and achieved statistical significance.

Group A study had a p value of .03, Group B's p value was .04.

Given only the above information which treatment is probably most effective?

Treatment B produced a much larger (but less reliable) effect size.

 

[marting]

Treatment B produced a much larger (but less reliable) effect size.

Yep, and the rub, and unknown in the example, is whether the trials were registered prospective or retrospective. The latter, with small N's is much more subject to publication bias since lots of groups may have conducted decent, but small, double blind trials and small ones are less likely to get published if not significant.

That's really the big problem with the current trials of Vit. D, Ivermectin, etc. They are pretty small. And reviewing the prospective registered trials show large percentages were started but never completed.