Question about statistical significance

[Rolfe]

OK, I should be able to do this but I can't. How's about some of you statistical geniuses give me a hand?

The variable is a simple yes/no. So for one trial, the probability of correct answer is 0.50. The probability of being correct both times in two trials is 0.25, right?

I need to know how many trials with what percentage success will give statistical significance (of correct answer not due to lucky guess) at p<0.001.

I've got a homoeopath begging me to test him, in public print. He seems to be claiming that homoeopathic preparations produce distinctive signs which can be recognised by a healthy person taking the remedy. (And indeed, according to homoeopathic theory this must be true - it's how they decide on the 'like', which will cure 'like'.) I'm trying to simplify the test down to a simple yes/no of, "Here you are, this is either the genuine remedy" - remedy used to be chosen by him in advance- "or a sham. Take it away, use whatever protocol you like, then come back when you're ready and tell me which."

Obviously this has to be repeated often enough to show statistical significance. What's the fewest number of trials we could do which would give a decent answer to this? If he got 18 right out of 20, would that reach p<0.001? Could we reasonably allow for fewer than 20 repetitions and still settle the matter conclusively? Does Randi agree to lesser degrees of significance being acceptable as a pass on a preliminary test?

It's taking a while to get this going, because so far the correspondence has been carried out in the Letters to the Editor section of a weekly professional journal, with a lag time of 2 or 3 weeks between sending a letter and it appearing in print. Right at the moment he's insisting that he can't do what I suggest because he needs a group of 10 people to take the test, he can't do it on his own. But in the same issue, my simplified test appears (changing from identifying the remedy given from a short list, to simply telling remedy of his choice from sham). Wires are still a trifle crossed.

Well, fine, if he needs a group of ten, let him rustle up a group of ten. There are no restrictions on how he figures out what it is, short of breaking into wherever the record is kept of what was actually given to him. NMR spectroscopy, X-ray diffraction, clairvoyance, dowsing, give it to his unfortunate patients and see if they recover, I don't give a monkey's.

I do need to know what standards I ought to hold him to in order to keep the chance of him winning by a run of lucky guesses acceptably low.

Rolfe.

 

[Mercutio]

18 out of 20 would be p<.0005. 17 out of 20, my table here says p=.001. That's assuming a .5 probability on a single trial.

What is the claimed level of accuracy? If he only claims to be able to be, say, 75% accurate, rather than 100% accurate, you have a different problem on your hands.

 

[Rolfe]

What is the claimed level of accuracy? If he only claims to be able to be, say, 75% accurate, rather than 100% accurate, you have a different problem on your hands.

That I don't know. I suspect we'll find out when I try him with "get it right 17 times out of 20". Could we settle for fewer trials? What about 13 right out of 15? I feel he's probably entitled to two errors, so how many trials would we need still to get p<=0.001?

Actually, his own suggested protocol is that I should be part of his group of 10! He's so convinced that I'd feel something that he wants me to experience it directly, even though he knows I think this is fairies at the bottom of the garden. This suggests that he imagines the potential accuracy to be rather good. Of course he won't ever have done a dry run to prove to himself that he can actually do it (and I'm not about to point out the four papers in the mainstream literature which got null results with a similar trial, this is just too much fun).

I'd like him to agree to a protocol which would be likely to be acceptable as a preliminary trial for the JREF challenge. Then we'll see if he believes in himself enough actually to make it an official application. I don't see why I should mess around with this nonsense unless it is an official attempt, quite frankly. And then we'd (I hope) get some more input about how best to ensure that the materials are prepared to the satisfaction of both sides, and to make sure that he can't break into wherever the real information is held and so on.

This one is actually better than the Benveniste/Ennis trial. When that one failed, they could just say "oh well, homoeopathy must work some other way then." Nowhere in homoeopathic dogma does it state that an ultradilute solution of IgE or histamine can degranulate basophils. However, homoeopathic dogma depends absolutely on this assertion that ultra-dilute remedies produce recognisable symptoms in healthy people. It's amazing one of them hasn't walked off with the million bucks years ago, really.

This really bugs me. They come over all sincere, and trot out the most preposterous tales of unrepeatable miracle cures and so on, and persuade everybody that even if they are completely barking nuts, they're sincerely deluded. Then when you point out to them that if what they claim is true, they could be a million bucks richer - suddenly there are all sorts of reasons why they won't go for the test. I tell you, if I really thought I could do this, I'd bust a gut to get tested! But no, I suppose we'll hear next that the money doesn't really exist, or Randi's legal restrictions are too onerous, or something. Sigh.

Rolfe.

 

[daver]

That I don't know. I suspect we'll find out when I try him with "get it right 17 times out of 20". Could we settle for fewer trials? What about 13 right out of 15? I feel he's probably entitled to two errors, so how many trials would we need still to get p<=0.001?

I get 16/18 trials. 8/10 or better is about 0.055, 16/18 about .00065.

 

[T'ai Chi]

OK, I should be able to do this but I can't. How's about some of you ...

Hi Rolfe, hope this will help. I calculated the various probabilities of getting more than r correct out of n, for various r and n combinations that you were or might be interested in, assuming the probability of "success" from trial to trial is .5. I bolded the r for each value of n where statistical significance at the .001 is first obtained.

n = 20
r > 15 yields .0206
r > 16 yields .0059
r > 17 yields .0012
r > 18 yields .0002
r > 19 yields .00002

n = 18
r > 13 yields .0481
r > 14 yields .0154
r > 15 yields .0037
r > 16 yields .0006
r > 17 yields .00007

n= 15
r > 10 yields .1508
r > 11 yields .0592
r > 12 yields .0175
r > 13 yields .0036
r > 14 yields .0004

n = 10
r > 5 yields .623
r > 6 yields .3769
r > 7 yields .1718
r > 8 yields .0546
r > 9 yields .0107
r = 10 yields .0009

In any case, especially the n = 10 size, I'd recommend repeating the entire test twice if possible.

I dunno, I'd also probably explain to everyone involved why getting significantly less than what is expected doesn't count.

 

[T'ai Chi]

What is the claimed level of accuracy? If he only claims to be able to be, say, 75% accurate, rather than 100% accurate, you have a different problem on your hands.

Hmm, interesting question.

If the claimed level of accuracy is less than 100%, how do we modify the statistical calculations to account for that? I don't believe we have to..but I'm not totally sure.

For example, with n = 20, if he claimed to be 100% accurate, we'd expect him to get 20 correct. If he claimed to be 75% accurate, we'd expect him to get .75*20 = 15 correct. Either way, we can test this using standard (I think) binomial calculations with the probability of "success" = .5. His claimed percentage correct doesn't have any bearing on the calculations, I don't think.

 

[pupdog]

I'd like to see some kind of dose-response graphs. Are dosages adjusted for body weight (1 molecule for adults, 1/16 molecule for children) or gender?
Is the choice of outcomes all or none, or is there any "some improvement" allowed.

 

[Stimpson J. Cat]

T'ai Chi,

If the claimed level of accuracy is less than 100%, how do we modify the statistical calculations to account for that? I don't believe we have to..but I'm not totally sure.

For example, with n = 20, if he claimed to be 100% accurate, we'd expect him to get 20 correct. If he claimed to be 75% accurate, we'd expect him to get .75*20 = 15 correct. Either way, we can test this using standard (I think) binomial calculations with the probability of "success" = .5. His claimed percentage correct doesn't have any bearing on the calculations, I don't think.

The claimed accuracy will essentially determine the number of trials needed. Consider the numbers you posted. With only 10 trials, a success of r>7 is only rejects the null-hypothesis at the 17.18% level. If the tester demands a 0.001 confidence, then clearly you are going to need more than 20 trials in order for a 75% success rate to give the necessary confidence level. Furthermore, the testee will (if he's smart) demand enough trials so that, assuming his success rate really is 75%, the actual threshold can be put far enough under 75% that he has a high chance of making it. After all, if the threshold is put right at 75%, then he only has a 50% chance of success!


Dr. Stupid

 

[T'ai Chi]

T'ai Chi,

The claimed accuracy will essentially determine the number of trials needed. Consider the numbers you posted. With only 10 trials, a success of r>7 is only rejects the null-hypothesis at the 17.18% level.

That makes sense, thanks Stimpy. I was trying to complexify things here I think.

 

[Vorticity]

...
I've got a homoeopath begging me to test him, in public print. He seems to be claiming that homoeopathic preparations produce distinctive signs which can be recognised by a healthy person taking the remedy. (And indeed, according to homoeopathic theory this must be true - it's how they decide on the 'like', which will cure 'like'.) I'm trying to simplify the test down to a simple yes/no of, "Here you are, this is either the genuine remedy" - remedy used to be chosen by him in advance- "or a sham. Take it away, use whatever protocol you like, then come back when you're ready and tell me which."
...

Wait a minute. Let me see if I have this straight:

He prepares either a placebo (plain water?) or a "genuine" remedy. Then you (or somebody) drinks it, and waits to see if there are any discernable physical effects.

CAREFUL!

How much do you trust this guy? Is there any oversight of his preperation? How do you know he won't simply slip some... uh... I don't know what... dilaudid? LSD? Any colorless, odorless, non-homeopathic thing, into the mixture to generate an "unmistakable physical response"?

If its really going to be a serious play for the JREF million, there better be some pretty good controls on what, exactly, is in the "active" batch. That's also probably a good idea for the health of the tester.

Can you nail down a definition of what constitutes "homeopathy" beforehand? I've encountered "homeopathic" preparations in drug stores that are NOT "homeopathic" in the sense we are familiar with. They contain large amounts of an active ingredient (think "1X" dilutions of something powerful). If a 1X, or even a 1C, dilution is classified as "homeopathy", then it may render homeopathy as trivially valid! After all, in even a standard medicine such as aspirin, the active ingredient is diluted by more than 1X (by volume).

Perhaps you should only allow homeopathic preparations in the test such that the dilution is so high that it can be shown that no molecules of the original active material remain in the mixture. After all: homeopaths claim that the mixture becomes more powerful the more you dilute it, so this should not be a problem.

 

[69dodge]

Re: Re: Question about statistical significance

Originally posted by T'ai Chi
n = 10
r > 9 yields .0107
r = 10 yields .0009

If r > 9, what else can it be but 10? So how can these yield different probabilities?

All your >'s should be >='s.

 

[Dilettante]

You want to make him to a

RANDOMIZED, DOUBLE-BLIND, CONTROLLED TRIAL.

"Control" is the placebo part.
"Randomized" means that the two groups are picked at random.
"Double-blind" means that you and he (and the guinea pigs) don't know which is which.

You will need a "ref" to randomly assign the groups. Give them "mixture 1" and "mixture 2" and ask them to put, say, one of them in 10 bottles, and the other in 10 bottles. Don't tell the ref which is homeopathic and which is placebo.

All the bottles get coded (no patient names, and nothing that tells which mix is in which bottle -- just "ab752" type stuff).
The ref doesn't know which patient gets what.
You don't know either.
Neither do the patients or the homeopath. Only the bottle numbers.
After X amount of time all the patients tell whether they got better.
Only then can you go to the ref and count how many were on homeopathy, and how many on placebo.

There must be references for this kind of thing online. It's the only way to be sure there's not some hidden correlation.
Of course, it doesn't tell you if the homeopath is secretly putting something extra in the mix. (Like the "pure rhino horn" that has Viagra in it.)

If the experimental design is right, the statistics are pretty simple. The most important thing is getting the design right.

 

[T'ai Chi]

Re: Re: Re: Question about statistical significance

If r > 9, what else can it be but 10? So how can these yield different probabilities?

All your >'s should be >='s.

Yes, you're correct, thank you. They should all be >='s.

 

[Beausoleil]

How do you have a placebo for a homeopathic mixture?

Surely all water contains everything diluted at the less than one molecule level.

Seriously, I've never understood it. It seems to be that only intentionally made homeopathic mixtures are supposed to count, even though we make them unintentionally all the time.

 

[Rolfe]

I've been away for the weekend, so I wasn't in touch.

Thanks to all who offered statistical insight into this, I'm a lot better informed now when it comes to trying to get the guy to agree what constitutes a valid test.

Please keep the suggestions coming, if you have any more.

Rolfe.

 

[Rolfe]

Re: Re: Question about statistical significance

Wait a minute. Let me see if I have this straight:

He prepares either a placebo (plain water?) or a "genuine" remedy. Then you (or somebody) drinks it, and waits to see if there are any discernable physical effects.

CAREFUL!

No, you don't have this entirely straight - sorry if I didn't explain it clearly.

This will definitely have to be a "no-molecules" ultra-dilute preparation - the guy in question has suggested 30C Belladonna. If a dilution under the Avogadro limit was used, clearly it wouldn't be eligible for any sort of prize.

Both the genuine remedy and the sham have to be prepared, certainly. Deciding how and who by is a matter for the later details. This is one reason I'm trying to get him to make a formal application, because it's the trickiest aspect of the whole thing. If there is a formal application in place, then this will all have to be gone into in detail, and it should be possible to interest a colleague in one of the universities in arranging or supervising the preparation of the materials.

However, it's done, obviously there has to be adequate supervision to make sure nothing tasty (or toxic or whatever) gets slipped into either preparation.

Then, the preparations must be kept secure, hopefully by trustworthy university colleagues, and issued to the applicant blind. My suggestion is to do it by coin toss - heads the real thing, tails the sham (or vice versa), and just give him what the coin says. Do it often enough for statistical significance to be determined, as we were discussing.

This is the only part which needs to be secure - the preparation and storage of the materials, the dispensing of same, and the records of what was actually dispensed on each occcasion. After that, since there is no restriction on the means the applicant may use to tell which he's been given, it doesn't much matter.

What he claims is that he can tell, by conducting a homoeopathic "proving", which he's got. Fine. Actually, I don't care what he does, but that's the method he claims. Originally, he seemed to be suggesting that even one person would experience enough to be able to tell, and I said, OK, do it yourself and get the million bucks.

The first slither-tactic was to declare that a single person wouldn't experience ALL the symptoms, so probably couldn't IDENTIFY the remedy alone. He said it had to be a group (ten, I think) who compared notes about what they experienced. He likened it to splitting a 120-piece jigsaw of a well-known picture randomly between ten people. One person might not know what the picture was from their 12 pieces, but if the group conferred they'd figure it.

I have two answers to this. First, I'm not asking him to identify the remedy, but simply to tell a known remedy, chosen by him, from a sham. So, he has either 12 pieces of the Mona Lisa, or 12 pieces of a blank jigsaw. Still can't do it? Surely! But second, if he really can't do it alone, then go find another 9 mates! No restriction on the method used, so that's fine. The composition of the group doesn't matter, it's entirely up to him, and if the supervision of the preparation has been good enough, then there should be no risk to anyone. Why he wants me to participate I don't know, but if he insists it might be fun. So long as I get my cut of the million bucks of course.

I don't know whether he'll insist on preparing the remedies himself, or will alow university pharmacology technicians to do it, but whichever, if there aren't sufficient sceptics breathing down the neck of the preparer for the entire process, I'm not playing.

Next wriggle prediction - Oh, the presence of sceptics stops the magic working and decouples the quantum entanglement.

Rolfe.

 

[Rolfe]

You want to make him to a
RANDOMIZED, DOUBLE-BLIND, CONTROLLED TRIAL.

No, I don't. You're getting confused between trying to prove that a homoeopathic medicine actually cures patients, and what the applicant in question is claiming.

I'm not making him do anything. He's claiming that it is possible to tell a homoeopathic remedy from a sham. That in itself wins the prize. If he can do it. All I'm trying to to is to arrange a protocol whereby he is given one or the other, in whatever quantities he wants, and is able to come back in his own time and declare which he has been given.

Frankly, I don't give a monkey's how he does it. He can do NMR spectroscopy, X-ray diffraction, dowse for it or ask John Edwards, I don't care. So long as he comes back with a decision.

What I expect him to do is to take it himself and see if some of the "proving" symptoms appear or not. That's what he claims is possible. He thinks it's so possible that even I could do it! However, he's the expert homoeopath, and if he wants the prize, I think he'd better do it himself.

His latest wriggle is implying that he needs a group of 10 people, not just one. OK, fine. Go get 9 mates. No need to supervise how he picks them, as there's no restriction on the methods he's allowed to use. If he asks me to be one of his 9 mates, well, so long as the preparation has been secure and rigorous as described above, maybe I will.

But that's it. Devise a way to issue either remedy or sham, randomly and securely, then let the applicant figure out which he's been given often enough to satisfy statistical significance. Any way he likes. Don't make it more complicated than need be.

Rolfe.

 

[Dilettante]

Will he be preparing the homeopathic solution?
If so, he might put a trace, detectable chemical inside without your knowing about it. Are you willing to watch him make the stuff?

 

[Rolfe]

Will he be preparing the homeopathic solution?
If so, he might put a trace, detectable chemical inside without your knowing about it. Are you willing to watch him make the stuff?

That's covered above. However he wants the stuff prepared, precautions will have to be agreed to ensure that it's done with scrupulous honesty. This is one of the reasons for wanting it to be an official try for the JREF prize - that will inevitably involve getting this sort of thing watertight.

I'm sort of betting with myself that he'll eventually declare that the presence of non-believers will decouple the quantum entanglement and so render any remedy so prepared useless, but maybe I'm giving him too much credit for ingenuity here. (Yes, I know the answer to that is why then do homoeopaths write prescriptions which might well be filled by non-believing pharmacists - it's sort of like a game of tennis.)

Rolfe.

 

[Vorticity]

...Yes, I know the answer to that is why then do homoeopaths write prescriptions which might well be filled by non-believing pharmacists...

Homeopaths can write prescriptions? In England?
That is rather alarming.
What can they write prescriptions for? Only homeopathic stuff, or real stuff as well? Do the pharmacists carry homeopathic "remedies"?