Merged Odds Standard for Preliminary Test

[Rodney]

I've heard several times on these forums about the "1 in one thousand usual MDC standard for a preliminary test." Unfortunately, there is nothing in the rules that mentions any such standard. Rather, there is only:

(1) The vague opening sentence to rule number 3. "We will consult competent statisticians when an evaluation of the experimental design, is required"; and

(2) Rule number 6. "In all cases, applicant will be required to perform a preliminary test either before an appointed representative, if distance and time dictate that need, or in a location where a member or representative of the JREF staff can attend. This preliminary test is to determine if the applicant is likely to perform as promised during a formal test, using the agreed-upon protocol. To date, no applicant has passed the preliminary test, and this has eliminated the need for formal testing in those cases. There is no limit on the number of times an applicant may re-apply, but re-application can take place only after 12 months have elapsed since the completion of the preliminary test."
See http://www.randi.org/joom/challenge-application.html

So what, exactly, is required to pass the preliminary test?

 

[GzuzKryzt]

Experience suggests that in most cases, one in 1000 chances are reflected in the protocols.

Since every claim seems to be different, however, the JREF has not set the standard at a certain probability mark. It depends on the claim.

 

[Cuddles]

I've heard several times on these forums about the "1 in one thousand usual MDC standard for a preliminary test." Unfortunately, there is nothing in the rules that mentions any such standard. Rather, there is only:

As has been explained many times before, it is not possible to have any universal standard because all claims are different. For example, the most recent test involved someone who claimed they could make someone with an empty bladder urinate using the power of their mind? How exactly would you set the odds for this? Obviously it would be paranormal, but the fact is tht you can't say it's 1000:1, or anything else. Either it happens or it doesn't, it is just not possible to work out the odds.

However, the actual odds are utterly irrelevant. The whole point of the tests is that it is not possible for someone to pass if they don't have whatever ability they claim. Since all applicants believe they do have their claimed abilities, the odds should make no difference to them at all. For example, if I claim that I can pick out a red card from a selection of face up cards, I don't care if the odds are 1000:1, 1000000:1 or anything else, because I know I can do it. Similarly, if I claimed to be able to do the same but with the cards face down, if I know I can do it, why would I care about the probability of doing it by random chance? I wouldn't.

In the end, the only people to whom the chance odds are in any way relevant are the JREF, since they have to ensure that a pass really is due to something other than chance.

 

[Rodney]

Experience suggests that in most cases, one in 1000 chances are reflected in the protocols.

Can you give me an example?

 

[GzuzKryzt]

Can you give me an example?

Sure.

Compare this to the Rosemary Hunter test. Hard to calculate the odds on the (prob)ability to make someone urinate, right?

 

[Rodney]

As has been explained many times before, it is not possible to have any universal standard because all claims are different. For example, the most recent test involved someone who claimed they could make someone with an empty bladder urinate using the power of their mind? How exactly would you set the odds for this? Obviously it would be paranormal, but the fact is tht you can't say it's 1000:1, or anything else. Either it happens or it doesn't, it is just not possible to work out the odds.

However, the actual odds are utterly irrelevant. The whole point of the tests is that it is not possible for someone to pass if they don't have whatever ability they claim. Since all applicants believe they do have their claimed abilities, the odds should make no difference to them at all. For example, if I claim that I can pick out a red card from a selection of face up cards, I don't care if the odds are 1000:1, 1000000:1 or anything else, because I know I can do it. Similarly, if I claimed to be able to do the same but with the cards face down, if I know I can do it, why would I care about the probability of doing it by random chance? I wouldn't.

In the end, the only people to whom the chance odds are in any way relevant are the JREF, since they have to ensure that a pass really is due to something other than chance.

I think your answer helps explain why there are so few non-delusional challengers for the prize. The idea that, for example, anyone can ALWAYS paranormally pick out a red card from a selection of face up cards is preposterous on the face of it (so to speak ). However, it may be possible for someone to pick out a red card at a significantly above chance rate, even if that rate is only 51-52% over thousands of trials. Would such a person apply for the prize, when there is not even a hint in the official rules of what level (s)he must perform at even to pass the preliminary test, let alone win the million dollars? The point is that, by not specifying an odds standard, serious challenges for the prize are discouraged. As a person involved in Ganzfeld experiments told me, the MDC is "so hedged and the criteria for success so arbitrarily set up and changeable at his [Randi's] whim that nobody will ever be able to pass his test."

 

[GzuzKryzt]

...
As a person involved in Ganzfeld experiments told me, the MDC is "so hedged and the criteria for success so arbitrarily set up and changeable at his [Randi's] whim that nobody will ever be able to pass his test."

Demonstrably false.

I would not expect though said person could be convinced by the facts which are the Challenge Rules.

Ask that person what s/he thinks about the Rosemary Hunter protocol. http://www.internationalskeptics.com/forums/showthread.php?postid=3011140#post3011140
Or about the Achau Nguyen protocol. http://www.internationalskeptics.com/forums/showthread.php?postid=1029440#post1029440
Or about the Angela Patel protocol. http://www.internationalskeptics.com/forums/showthread.php?postid=1048225#post1048225

 

[Rodney]

Sure.

Thanks for the link to the Carina Landin thread. I take it that she is going to be retested soon? Suppose she gets 15 right out of 20 this time; that would be 27 right out of 40 in the two tests combined. Would any thought be given to allowing her to do a longer test over a period of time, perhaps encompassing as many as 100 trials?

Compare this to the Rosemary Hunter test. Hard to calculate the odds on the (prob)ability to make someone urinate, right?

Certainly harder, although statistics could be gathered on urination frequency.

 

[Czarcasm]

Good examples would be claims of telekinetics, teleportation or levitation. How would one figure out the odds of these paranormal claims?

 

[juryjone]

Perhaps your Ganzfeld buddy will lower himself to apply for the test, and we can find out what kind of criteria are required? As it is, all the applicants for the MDC have stated "I can do such and such to this level of accuracy" - usually 100% accuracy. It's Randi who then suggests that maybe they ought to try for less accuracy, such as the dowsing trials where the subject has to pick the correct container out of ten seven times.

It is obvious that the number of trials is dependent on the claim. It has been stated before on this very thread. Is it really so difficult to understand?

Might I also suggest that you are being disingenuous in asking this question, given the number of threads that have been started on this very subject over the 2+ years you've participated in the forum?

 

[GzuzKryzt]

Good examples would be claims of telekinetics, teleportation or levitation. How would one figure out the odds of these paranormal claims?

Or Prophet Yahweh with his summoning of UFOs.

 

[Moochie]

Perhaps your Ganzfeld buddy will lower himself to apply for the test, and we can find out what kind of criteria are required? As it is, all the applicants for the MDC have stated "I can do such and such to this level of accuracy" - usually 100% accuracy. It's Randi who then suggests that maybe they ought to try for less accuracy, such as the dowsing trials where the subject has to pick the correct container out of ten seven times.

It is obvious that the number of trials is dependent on the claim. It has been stated before on this very thread. Is it really so difficult to understand?

Might I also suggest that you are being disingenuous in asking this question, given the number of threads that have been started on this very subject over the 2+ years you've participated in the forum?

Perhaps he's waiting for the "right" answer?

M.

 

[Rodney]

Demonstrably false.

I would not expect though said person could be convinced by the facts which are the Challenge Rules.

Ask that person what s/he thinks about the Rosemary Hunter protocol. http://www.internationalskeptics.com/forums/showthread.php?postid=3011140#post3011140
Or about the Achau Nguyen protocol. http://www.internationalskeptics.com/forums/showthread.php?postid=1029440#post1029440
Or about the Angela Patel protocol. http://www.internationalskeptics.com/forums/showthread.php?postid=1048225#post1048225

I think all of the above protocols were reasonable; still, there remains the glaring deficiency in the official rules that no odds standard is specified, either for the preliminary test or for the formal test that theoretically could result in the prize being awarded. At a minimum, something along the lines of the following should be added to the Challenge Rules:

"An applicant must pass a preliminary test, in which the general criterion for success will be that the applicant must perform at significantly above the chance level. In tests where the odds of success can be easily calculated -- such as numbers guessing -- the applicant must perform at least at the P=.001 level; that is, the odds must be only one in one thousand that the applicant could have achieved that performance level by random chance. (However, if the applicant achieves a lesser, but above chance, performance level in a limited number of tests -- for example, if the applicant performs at the P=.01 level in 20 trials -- the preliminary test may be extended on a different day or days to include more trials.) If the applicant passes the preliminary test, a final test will be administered, in which the performance level must meet a significantly more stringent criterion for the million dollar prize to be awarded. In tests where the odds of success can be easily calculated, the applicant must perform at least at the P=.000001 level; that is, for the prize to be awarded, the odds must be only one in one million that the applicant could have achieved that performance level by random chance."

 

[Rodney]

Good examples would be claims of telekinetics, teleportation or levitation. How would one figure out the odds of these paranormal claims?

In those cases, I would think a single demonstration would suffice to pass the preliminary test.

 

[Rodney]

Perhaps your Ganzfeld buddy will lower himself to apply for the test, and we can find out what kind of criteria are required?

The problem with that is the hassle the person will have to go through to find out, in all probability, that the Randi Foundation considers a Ganzfeld protocol too time-consuming to be tested.

As it is, all the applicants for the MDC have stated "I can do such and such to this level of accuracy" - usually 100% accuracy. It's Randi who then suggests that maybe they ought to try for less accuracy, such as the dowsing trials where the subject has to pick the correct container out of ten seven times.

It is obvious that the number of trials is dependent on the claim. It has been stated before on this very thread. Is it really so difficult to understand?

No, what's difficult to understand is why there is no odds standard in the Challenge Rules.

Might I also suggest that you are being disingenuous in asking this question, given the number of threads that have been started on this very subject over the 2+ years you've participated in the forum?

And yet, after all this time, no progress has been made: There is still no odds standard in the Challenge Rules, despite many posters here claiming that there is a P=.001 standard for the preliminary test.

 

[Klimax]

I think,that odds are not part of rules,simply because each challenger has his own perception(each claim is bit different) and since this is part of protocol,which is discussed as well in communication with JREF,I doubt it would help any better.

Those who won't apply because "Randi changes the rules of experiment" will then say the rule is arbitrary strict,so nobody will pass...
That is why it is part of negotiation and not written in rules in hard numbers.

Just some of my (Euro) cents.

 

[juryjone]

Perhaps he's waiting for the "right" answer?

M.

I think you may be right.

I think all of the above protocols were reasonable; still, there remains the glaring deficiency in the official rules that no odds standard is specified, either for the preliminary test or for the formal test that theoretically could result in the prize being awarded. At a minimum, something along the lines of the following should be added to the Challenge Rules:

"An applicant must pass a preliminary test, in which the general criterion for success will be that the applicant must perform at significantly above the chance level. In tests where the odds of success can be easily calculated -- such as numbers guessing -- the applicant must perform at least at the P=.001 level; that is, the odds must be only one in one thousand that the applicant could have achieved that performance level by random chance. (However, if the applicant achieves a lesser, but above chance, performance level in a limited number of tests -- for example, if the applicant performs at the P=.01 level in 20 trials -- the preliminary test may be extended on a different day or days to include more trials.) If the applicant passes the preliminary test, a final test will be administered, in which the performance level must meet a significantly more stringent criterion for the million dollar prize to be awarded. In tests where the odds of success can be easily calculated, the applicant must perform at least at the P=.000001 level; that is, for the prize to be awarded, the odds must be only one in one million that the applicant could have achieved that performance level by random chance."

Please explain to me how all this language, which is not applicable to anyone who has applied to the challenge thus far, will make the rules clearer and make those who have refused to take the challenge more amenable.

The rules in a nutshell are:
1) Claimant makes claim.
2) JREF works with claimant to develop a controlled test of that claim that both the JREF and the claimant can agree to.

Odds don't enter into it. If someone said that odds of 1:1000 are needed for the preliminary test, that number was yanked out of their nether region, since, as you stated, those are not part of the rules.

My guess (and I stress the word guess) is that the JREF would not enter into a test of a claim that only falls marginally outside the realm of chance. It would require too much testing, for insignificant results.

Each claim is analyzed on its own merits, and no "odds standard" or p-value should be shoehorned into the rules to ensure that there will be more obstacles in the way of claimant and JREF agreeing on a test.

 

[Rodney]

Please explain to me how all this language, which is not applicable to anyone who has applied to the challenge thus far, will make the rules clearer and make those who have refused to take the challenge more amenable.

It might encourage someone to apply who thinks (s)he may be able to demonstrate the paranormal, but who believes that a short test will be insufficient for that demonstration.

The rules in a nutshell are:
1) Claimant makes claim.
2) JREF works with claimant to develop a controlled test of that claim that both the JREF and the claimant can agree to.

Odds don't enter into it. If someone said that odds of 1:1000 are needed for the preliminary test, that number was yanked out of their nether region, since, as you stated, those are not part of the rules.

Odds don't enter into it??? Again, only a delusional person is going to believe that (s)he can always determine through paranormal means a number that has been written on a piece of paper, the color or suit of a face-down playing card, etc.

My guess (and I stress the word guess) is that the JREF would not enter into a test of a claim that only falls marginally outside the realm of chance.

I'm quite confident that you are correct about the JREF's viewpoint.

It would require too much testing, for insignificant results.

It seems to me that herein lies the problem with the Challenge. If, for example, an applicant could consistently determine with 54% accuracy whether a face-down playing card was red or black, even 100 trials would be woefully insufficient to prove that, because 54 hits in 100 trials could easily be attributable to chance (P=.24). However, if the applicant could continue to perform at that level over about 3500 trials, (s)he would meet a P=.000001 criterion. So, while the Challenge is useful for disproving claims of "paranormal powers that never fail", it does not appear to be useful for disproving alleged powers that operate at a much lower success ratio.

Each claim is analyzed on its own merits, and no "odds standard" or p-value should be shoehorned into the rules to ensure that there will be more obstacles in the way of claimant and JREF agreeing on a test.

An odds standard would simply let applicants know that there is a standard, rather than them understandably thinking that it's a moving target.

 

[GzuzKryzt]

...
No, what's difficult to understand is why there is no odds standard in the Challenge Rules.

And yet, after all this time, no progress has been made: There is still no odds standard in the Challenge Rules, despite many posters here claiming that there is a P=.001 standard for the preliminary test.

There is no standard because there are no standard claims. The claims vary, the protocols vary, hence the odds vary. It's that simple.

As Cuddles already said correctly, the odds do not matter. The demonstration matters: "Do what you claim to be able to do and win a million dollars."

A stipulation of a P=.001 standard for the preliminary test of summoning UFOs or teleportation/psychokinesis would be irrelevant, absurd even, wouldn't it? Peter Hubinsky's claim comes to mind.



I have to admit though, Rodney, I had the same question at the beginning of my lurking in 2003. After reading every post in the Challenge Application subforum, things cleared up for me.

 

[69dodge]

The idea that, for example, anyone can ALWAYS paranormally pick out a red card from a selection of face up cards is preposterous on the face of it (so to speak ). However, it may be possible for someone to pick out a red card at a significantly above chance rate, even if that rate is only 51-52% over thousands of trials.

I think you misunderstood the example. It's easy to pick red cards consistently if the cards are face up.

As a person involved in Ganzfeld experiments told me, the MDC is "so hedged and the criteria for success so arbitrarily set up and changeable at his [Randi's] whim that nobody will ever be able to pass his test."

Even if the paranormal exists, I assume the implication is? Otherwise, of course, it's perfectly reasonable that no one be able to pass.

Many tests have already been conducted, which were agreeable to their participants. So, it is clear that, sometimes at least, Randi does stop changing his criteria long enough for a test to proceed.

In any case, I can't quite figure out what your complaint is. Is it that Randi is not willing to spend an unlimited amount of time on any one applicant? Or, perhaps, that he doesn't give, in the official rules, an indication of how much time he is willing to spend?