Merged Odds Standard for Preliminary Test

[pavel_do]

I have to disagree with you on Pavel's behalf. Pavel is not deliberately complicating the protocols, Pavel clearly did not know how to put one together nor did he know what was required to meet JREF's 1:1000 odds pass rate.
He stated this in a number of posts prior to applying to the Challenge.

It's quite clear that he came to the Forum to determine at what level he needed to perform his skill to, to pass the Challenge. Others who now claim that JREF is doing hard by him overlook the point that he never claimed a performance % but asked JREF what the threshold % was.
Again, in his defence, Pavel has done some semi-blinded tests at Forum member's urgings and shared them in the Forum. We (surprise, surprise) were quick to demonstrate to him that his then current (pre application) success rate was not better than chance (his 70% claim) and that he'd have to perform better to pass the Challenge.

I haven't seen any of Pavel's posts that indicate he is doing this in anything but good faith, so I have to disagree with your pessimism here. I do think that he will fail, but have confidence in him and the Forum members helping him that he'll end up getting tested.

Just to reiterate, since it is relevant to the thread.

Pavel came to the Forum to find out what success rate was required for him to pass the Challenge.

The accusations that JREF is forcing him to attempt something at a level of success that he didn't claim he could do are unfounded.

Thank you.
just one comment here.. when I really tried to find out the odds and things, by emailing to JREF.. I was refused any answers , reasoning that I need to apply firs and after that we start any negotiation.. even when I have asked.. if I claim sirtain % of a minimum rate ( that JREF ask to state in application and description of the claim) even that was ignored..
here is the letter and the PM to Remiev that I have sent a few days before I sent my application to JREF.
"Dear Alison,



Thank you very much for the reply,

Here is the text that I will attach to my application maybe it will be edited slightly but mainly will be the same. Please review it as to avoid the application to be returned to be due to some insignificant mistakes or disagreements from the side of JREF. As to the accuracy, I understand what you were saying, the only thing as the number of the successful photos for me will depend on the finale agreed protocol. As it still can be set of 20 photos where 5 pulled out and I would say I will be right with 3 minimum or let say set of 20 pars where I would have to identify just one photo from each pair, that makes numbers different. Can I state in my claim the ODDS or a % instead of definite number? As JREF demand 1 to 1000 odds to be bitten irrelevantly from the way, it performed. So I would like to claim ODDS as well. In my opinion it will be a success if I will beat say 150 odds as it is it self way more to the chance, taking to the considerations that it will be not 100 runs and just a few like in case with 20 pairs set I would prefer 3-5 runs by 20 sets. Or with 20 photos where 5 will be taken I would prefer have 3 runs only. Will it be fine for the claim as the accuracy to state 70% minimum from my side?


If so, then the following text ( still might be slightly edited) will be attached to the application form together with media coverage (photo copy of the news papers with the articles about me) and the letter signed by 2 professors who tested me and witnessed the results.


"I am Pavel, psychic clairvoyant. As a part of my gift, I have the ability to identify photographs that are sealed in double envelopes without seeing them first, just by holding the envelope between my hands and concentrating for a few minutes. I would prefer to use sets of photos that are completely different from each other, for example, a horse, the statue of liberty, ship, mask etc. The photos will be known to me and the set that is used for testing will be made of the same pictures that I usually practice with , and will be provided to the JREF beforehand together with the name for each photo. However, no one can speak while I perform and there should be no distraction of any kind. For that reason, I prefer that my eyes are covered with a mask, and I will use earplugs, to eliminate light and sound distractions. It is important that in the room where the test will be held, there should be NO photos or pictures of any kind on the walls or anywhere near me. I will need a small break between the runs if there will be a few runs of the testing. I need to come 1 hour before the test, to the place where it will be handled (as I need to get used to place and the people around to be calmer and make sure that I will be comfortable in my place during the test). Such an ability I can perform with the accuracy of a minimum 70% with the success rate significantly better than random chance. ”

Reagards,

Pavel

P.s.

I would appreciate your answer as I need to send my application A.S.A.P a specially if it take more time. Now, As I would like to have my test in US and while I am here, I am here till the September 1 the latest… due to protocol negotiation etc it will take time.. so please just answer last letter and I can send my application."

here is the reply....

"
Re: this is PAVEL

--------------------------------------------------------------------------------

No, Pavel. I will not read and reply to it. As previously stated, the application and associated materials must be received before I will look over any protocol information. No matter what you include with the Challenge Application, it will not be your full protocol. That is negotiated over time.

Please submit the application via postal mail, and do not send any more queries via e-mail or PM.

Thank you "

 

[petre]

I'm not talking about a change in the protocol, I'm talking about the odds standard that must be met. I've always been under the impression that the formal test would require a higher odds standard than the preliminary test. For example, in Pavel's case, he might have to get 30 of 40 to pass the preliminary test, but 60 out of 80 to pass the formal test.

The performance requirement for success is as much a part of the protocol as any other part. Therefore the odds requrement cannot be increased for the final test (well, without agreement from both parties, and it would hardly seem in the applicant's interest to raise the bar for a chance at a $million).

You were expecting something like: "Very good job passing the preliminary. Now for the formal test we require you to succeed 60 million out of 80 million trials! Muhahahaha!" ? The contract says the formal test WILL proceed, not conditional to the applicant agreeing to new performance requirements. Insisting on any change to the protocol that the applicant does not agree to before administering a formal test would be a breech of the contract as worded.

Of course since it is an agreement between two parties they certainly could agree to a different standard between the two tests in the original protocol. Maybe the best Pavel can do in one day is a test with 1:500 chance of success (by luck alone) and is willing to take on 1:2000 odds in a longer formal test if the JREF will agree to this relaxation of unwritten standard (or perhaps 1:2500 odds even, just performing the preliminary test twice more for the formal test). The JREF might find such a suggestion agreeable, since there is still a high probability of by-luck failure in the preliminary and the formal would never need to be given.

Having the odds standard unwritten allows this flexability. In general, once a rule is in the formal application, it is indeed a RULE. No exceptions to the explicit rules have ever been granted to an applicant (to my knowledge). Randi has occasionally offered to waive the preliminary test for certain high-profile paranormal claimants, but even that provision may be in the rules (I haven't read it over fully recently).

 

[Rodney]

The performance requirement for success is as much a part of the protocol as any other part. Therefore the odds requrement cannot be increased for the final test.

But isn't only the preliminary -- and not the final -- test being negotiated right now for Pavel?

 

[steenkh]

What do you base the above on?

From the official challenge page:

This preliminary test is to determine if the applicant is likely to perform as promised during a formal test, using the agreed-upon protocol.

There is only a single protocol. There is not one protocol for the preliminary test and another one for the final test.

 

[steenkh]

just one comment here.. when I really tried to find out the odds and things, by emailing to JREF.. I was refused any answers , reasoning that I need to apply firs and after that we start any negotiation.. even when I have asked.. if I claim sirtain % of a minimum rate ( that JREF ask to state in application and description of the claim) even that was ignored..

Is this not because you are not stating what your ability is. You are asking for odds that you will have to beat, which will vary wildly according to your stated ability. If you had said from the beginning: "I can identify the pictures with an accuracy of 70%", the JREF might have had a chance to work out how many tests you would have to do at what success rate, but instead you are asking them something that is impossible to answer.

You should only worry about your own ability, and the JREF can worry about the odds for a false positive.

 

[steenkh]

But isn't only the preliminary -- and not the final -- test being negotiated right now for Pavel?

As I said in my previous post: there is only a single protocol, so this protocol will also be the protocol for the final test. However, the final test may still differ.

The Challenge FAQ states

5.2 What happens between the preliminary test and the official test?

The protocol itself will not be changed, and neither will any of the documents you and the JREF have agreed upon. The final test may be longer, or require more conclusive results through more sets of the test to ensure that the preliminary test was not a fluke.

 

[steenkh]

Being required to pass the same test twice in a row results in the higher odds standard. That's the way I have always read that requirement.

As my quote from the FAQ in my previous post shows, James Randi thinks that the final test may require different odds.

 

[petre]

As I said in my previous post: there is only a single protocol, so this protocol will also be the protocol for the final test. However, the final test may still differ.

The Challenge FAQ states

I'm curious how the distinction is made then between the agreed-upon protocol and the performance requirement.

 

[drkitten]

I'm curious how the distinction is made then between the agreed-upon protocol and the performance requirement.

Reason and good faith, applied jointly.

If your claim is that you can perform at the 80% accuracy level, and the protocol is negotiated to require you to perform at the 60% accuracy level to pass the preliminary, it is not unreasonable to also require you to perform at the 60% accuracy level to pass the final.

The difference is that you will need to perform more repetitions of an identical task. Instead of needing to get 6 right out of 10, you may need to get twelve right out of 20 (or whatever the numbers work out to be).

 

[Jackalgirl]

"At JREF, we offer a one-million-dollar prize to anyone who can show, under proper and feasible observing conditions, evidence of any paranormal, supernatural, or occult power or event."?

Linda

Simpler is better, of course. But may I add that Randi already stipulates a control for time and effort:

I, James Randi, through the JREF, will pay US$1,000,000 [One Million Dollars/US] to any person who can demonstrate any psychic, supernatural or paranormal ability under satisfactory observing conditions.

(Emphasis mine.) The conditions have to be satisfactory to both JREF and the claimant, as is made clear later by the "mutually agreeable" protocol stipulation.

Could JREF go into nauseating detail about what "satisfactory" means? I suppose they could, but as you can see, trying to do so (and cover all the possibilities) gets into a highly complicated what-if game. It's simpler -- and better, in my opinion -- to simply say "satisfactory" and "mutually agreeable" and work that out via discussion with the claimant, especially since the actual definition of what all of that means hangs from the actual claim. Without knowing what the claim is, trying to define the parameters of what is and isn't acceptable is very difficult.

 

[Jackalgirl]

It is not relevant who prescribed the trials or what the total number is. I'm just pointing out that your proposed accomodation introduces an error.

Linda

I just got finished reading Mr. Randi's Flim-Flam! -- he calls what you're calling "early stopping" "optional stopping". He says:

Briefly stated, if the subject is allowed to stop whenever he or she wants, there is no value to the experiment, since the subject can stop or be stopped when ahead, and the total result is a win, regardless of what would have happened had the test continued. For this reason the experiment must have an announced number of trials determined firmly in advance, as was done. But optional stopping can also be optional continuing. It's the same problem. If results don't look too good...it is easy to throw in anther few dozen trials to see if we can get ahead before stopping.

(Flim-Flam!, Randi, Prometheus Books, Buffalo New York, 1982, p 236)

Given this, I think it's safe to say that Pavel will have to specify a specific number of trials and he will have to complete them all, to the level of accuracy that he claims. (In other words, JREF will not allow early optional stopping or optional continuing.) If that set of trials does not meet the 1:1000 requirement, he'll have to do more until that requirement is met.

I really hope that JREF begins to dialogue with Pavel and Startz via email in a robust and efficient manner. I'm really interested in seeing how this develops.

 

[Rodney]

I just got finished reading Mr. Randi's Flim-Flam! -- he calls what you're calling "early stopping" "optional stopping". He says:

(Flim-Flam!, Randi, Prometheus Books, Buffalo New York, 1982, p 236)

Given this, I think it's safe to say that Pavel will have to specify a specific number of trials and he will have to complete them all, to the level of accuracy that he claims. (In other words, JREF will not allow early optional stopping or optional continuing.) If that set of trials does not meet the 1:1000 requirement, he'll have to do more until that requirement is met.

Two points: (1) Beating odds of 1:1000 by random chance is very unlikely, even if early stopping is allowed; (2) My proposal is not to allow early stopping, but to allow the preliminary test to continue if Pavel (or anyone else) initially performs significantly above chance, but short of the 1:1000 level.

I really hope that JREF begins to dialogue with Pavel and Startz via email in a robust and efficient manner. I'm really interested in seeing how this develops.

Yes, and it's hard to understand why the JREF is moving at such a snail's pace when Pavel says that he is ready to take the preliminary test.

 

[EHocking]

Been there, done that. On May 10, 2008, I e-mailed the JREF as follows:

"I recently initiated the following thread on the Million Dollar
Challenge Forum --
http://www.internationalskeptics.com/forums/showthread.php?p=3692318#post3692318

"What I argue on that thread is that: (a) In tests where the odds of
success can be readily calculated, it is unclear what odds standard
must be met; ...

Just trying to cut back to the OP of this thread, perhaps it would be useful to review those protocols that have been successfully negotiated to determine how the odds of success are determined. The application list is at the end of this post.

Of the 11 application, the required success rate to pass the Challenge were all given by the applicants and JREF accepted every single one of the applicant's success rates. One exception was that JREF suggested an 80% score instead of 100% (Corey).

So, for the successfully negotiated protocols so far, JREF has accepted the Applicants' claimed success rates or less for the Challenge.

The spread of odds is large.

Of the 11 Applicants 4 were "self evident", that is a phenomenon occurred or didn't (i.e. yes/no) and odds/statistics weren't discussed.
Of the remaining 7, two were to be tested at 2:100 odds, 2 at 1:10,000 odds, 1 at 1:1,000,000+ odds and 2 at greater than 1:3,000,000.

This clearly demonstrates to me that the odds for each Challenge is determined by the Applicant's claim - just as the Challenge Rules declare.

Setting an absolute statistical success rate is as important to the Challenge procedure as it is impractical.

Each claim determines it's own success rate.
Just so long as it fits the Applicant's claimed success rate and JREF is satisfied that their claim is sufficiently better than that which would occur due to random chance(i.e. "paranormal"), taking the claimant's proposed success rate is obviously quite sufficient.

Carina Landin
Claim : to identify dead person's identity for a sitter
Success Rate : 80% by applicant
Protocol : guess gender of letter writer - 16/20.
Odds : <1:10,000

Ian Conger (not tested)
Claim : determine 5 letter words "sent" to Oiuja board
Success Rate : 2 of 3 words by applicant
Odds: 1: 62,500,000,000 (by applicant)

Achau Nguyen
Claim : psychically send words to receiver
Success Rate : 19+/20 by applicant
Odds: better than 1:1,000,000

Hans Peter Borer
Claim : Dowse mobile phone from 10 boxes
Success Rate : implied 100% by applicant
Protocol : 13 trials by GWUP tester
Odds: 1:100 (at 6 from 15)

Angela Patel
Claim: dowse a person's address from A-Z
Success rate: 1 from 1 - 100% from the applicant
Agreed : 3 from 3 (JREF suggested 5).
Odds : none

Cameron Johnson
Claim: psychically transmit 10 cards to a receiver
Success rate: 10/10 applicant's claim (from previous test)
Odds: 1:3,600,000 (determined by applicant)
Tested by carolina skeptics

Paul Carey (not tested)
Claim: Transmit phrases psychically.
Success rate: claimed 100%, Jref suggested 4/5
Odds: 1:100

Yellow Bamboo (not tested)
Claim: Knock down an attacker without touching them.
Success Rate: yes/no i.e. 100% by applicant
Odds: none

Jim Dunn
Claim: Prevent hospital deaths over 24+hr period
Success Rate : y/n i.e. 100% by applicant
Odds: none

Russell Shipp (not tested by JREF)
Claim : Telekinesis. Spin a suspended object.
Success Rate : yes/no i.e. 100% by applicant
Odds: none

James Blunt (not tested by JREF)
Claime : identify 5 materials in 5 bags by dowsing, 3 trials
Success Rate : 100% by applicant
Odds: in the region of 1:10,000?

 

[GzuzKryzt]

...
Yes, and it's hard to understand why the JREF is moving at such a snail's pace when Pavel says that he is ready to take the preliminary test.

Agreed. That is hard to understand, even with the usually limited resources available.

Just trying to cut back to the OP of this thread, perhaps it would be useful to review those protocols that have been successfully negotiated to determine how the odds of success are determined. The application list is at the end of this post.

Of the 11 application, the required success rate to pass the Challenge were all given by the applicants and JREF accepted every single one of the applicant's success rates. One exception was that JREF suggested an 80% score instead of 100% (Corey).

So, for the successfully negotiated protocols so far, JREF has accepted the Applicants' claimed success rates or less for the Challenge.

The spread of odds is large.

Of the 11 Applicants 4 were "self evident", that is a phenomenon occurred or didn't (i.e. yes/no) and odds/statistics weren't discussed.
Of the remaining 7, two were to be tested at 2:100 odds, 2 at 1:10,000 odds, 1 at 1:1,000,000+ odds and 2 at greater than 1:3,000,000.

This clearly demonstrates to me that the odds for each Challenge is determined by the Applicant's claim - just as the Challenge Rules declare.

Setting an absolute statistical success rate is as important to the Challenge procedure as it is impractical.

Each claim determines it's own success rate.
Just so long as it fits the Applicant's claimed success rate and JREF is satisfied that their claim is sufficiently better than that which would occur due to random chance(i.e. "paranormal"), taking the claimant's proposed success rate is obviously quite sufficient.

Carina Landin
Claim : to identify dead person's identity for a sitter
Success Rate : 80% by applicant
Protocol : guess gender of letter writer - 16/20.
Odds : <1:10,000

Ian Conger (not tested)
Claim : determine 5 letter words "sent" to Oiuja board
Success Rate : 2 of 3 words by applicant
Odds: 1: 62,500,000,000 (by applicant)

Achau Nguyen
Claim : psychically send words to receiver
Success Rate : 19+/20 by applicant
Odds: better than 1:1,000,000

Hans Peter Borer
Claim : Dowse mobile phone from 10 boxes
Success Rate : implied 100% by applicant
Protocol : 13 trials by GWUP tester
Odds: 1:100 (at 6 from 15)

Angela Patel
Claim: dowse a person's address from A-Z
Success rate: 1 from 1 - 100% from the applicant
Agreed : 3 from 3 (JREF suggested 5).
Odds : none

Cameron Johnson
Claim: psychically transmit 10 cards to a receiver
Success rate: 10/10 applicant's claim (from previous test)
Odds: 1:3,600,000 (determined by applicant)
Tested by carolina skeptics

Paul Carey (not tested)
Claim: Transmit phrases psychically.
Success rate: claimed 100%, Jref suggested 4/5
Odds: 1:100

Yellow Bamboo (not tested)
Claim: Knock down an attacker without touching them.
Success Rate: yes/no i.e. 100% by applicant
Odds: none

Jim Dunn
Claim: Prevent hospital deaths over 24+hr period
Success Rate : y/n i.e. 100% by applicant
Odds: none

Russell Shipp (not tested by JREF)
Claim : Telekinesis. Spin a suspended object.
Success Rate : yes/no i.e. 100% by applicant
Odds: none

James Blunt (not tested by JREF)
Claime : identify 5 materials in 5 bags by dowsing, 3 trials
Success Rate : 100% by applicant
Odds: in the region of 1:10,000?

Nice work, EHocking.



Could someone calculate the odds for the Yellow Bamboo claim? I know, but anyway.

 

[EHocking]

Could someone calculate the odds for the Yellow Bamboo claim? I know, but anyway.

Grrr, one of the points of my post was to show Rodney that in some applications odds are irrelevant.

The claimant either demonstrates a phenomenon or doesn't.
Odds in a claim of that nature are irrelevant.

 

[EHocking]

...Yes, and it's hard to understand why the JREF is moving at such a snail's pace when Pavel says that he is ready to take the preliminary test.

One of the reasons, IMHO, is that Pavel is too readily lead into complicating a protocol that should have been straightforward. Initiall because he was not clear on what he could do, and then got into detailed protocol discussions without a clear goal in mind.

As can be seen from the successful protocols, when the applicant clearly sets out what they can do the process really only bogs down on logistics.

I still suggest that Pavel goes back to his original claim (per my previous post) and simplifies the protocol. It is a simple one: guess one or two cards from a choice of three, conducted ten times, goal being 70%.

 

[GzuzKryzt]

Grrr, one of the points of my post was to show Rodney that in some applications odds are irrelevant.

The claimant either demonstrates a phenomenon or doesn't.
Odds in a claim of that nature are irrelevant.

I know. Still, it seems an interesting challenge for a stat wiz to calculate the odds for e.g. Prophet Yahweh's claim.

 

[Coveredinbeeees]

Just trying to cut back to the OP of this thread, perhaps it would be useful to review those protocols that have been successfully negotiated to determine how the odds of success are determined. The application list is at the end of this post.

Carina Landin
Claim : to identify dead person's identity for a sitter
Success Rate : 80% by applicant
Protocol : guess gender of letter writer - 16/20.
Odds : <1:10,000

Hans Peter Borer
Claim : Dowse mobile phone from 10 boxes
Success Rate : implied 100% by applicant
Protocol : 13 trials by GWUP tester
Odds: 1:100 (at 6 from 15)

You might want to recheck your calculations here. I Had a quick look at 2 of these and found mistakes.

For Carina Landin, the odds of getting at least 16 out of 20 genders correct purely by chance would be 0.0059 or 1:170.

For Hans Peter Borer, his protocol was actually to find which of the 10 boxes held the phone at least 7 times in 13 tests. Odds of achieving this by chance alone are 0.0001 or 1:10,000

Here's a handy tool if you don't want to make your own.

It was interesting to see all of the tests laid out this way. I read through the applications section when I first came across these forums and it was fun to be reminded of some of the highlights, or lowlights, of the challenge.

Cheers,

'Beeees

 

[EHocking]

You might want to recheck your calculations here. I Had a quick look at 2 of these and found mistakes.

I had made a mistake, I seem to have mixed the two examples when typing the list out. Where possible, I used this set of tables as a reference, to get a feel for the odds.

For Carina Landin, the odds of getting at least 16 out of 20 genders correct purely by chance would be 0.0059 or 1:170.

Indeed, I was surprised that, from the above tables, 16/20 with a 1 from 2 choice is to be expected to occur purely by random chance at odds of 1:100.

The point was that JREF were satisfied that 16/20 (the 80% of the claimant) was sufficient for this applicant to pass the Preliminary Challenge.

For Hans Peter Borer, his protocol was actually to find which of the 10 boxes held the phone at least 7 times in 13 tests. Odds of achieving this by chance alone are 0.0001 or 1:10,000

Here's a handy tool if you don't want to make your own.

See previous explanation. The tables I referenced support that range of odds.

It was interesting to see all of the tests laid out this way. I read through the applications section when I first came across these forums and it was fun to be reminded of some of the highlights, or lowlights, of the challenge.

Cheers,

'Beeees

I compiled a spreadsheet of the entire list (about 6months out of date now) to determine just how many applicants get to a final protocol and the reasons for not getting there. It's actually difficult to analyse statistically because of the number of applications that aren't or don't need to be determined by odds compared to random chance. A number of them are "it either happens or it doesn't".

Point I was trying to get across to the OP is that his fixation on JREF stating a standard "pass" rate is quite irrelevant and is a strawman constructed by him to discredit the Challenge.

I will admit though, I had always thought that the Preliminary stage's "pass" threshold was 1:10,000 and the Final was 1,000,000. It may well have been when the Challenge was only for a $10,000 prize, but I've not been able to find evidence for that and quickly reviewing the Challeng Application section will show that it's not applicable in quite a number of applications to the Challenge.

 

[Coveredinbeeees]

I had made a mistake, I seem to have mixed the two examples when typing the list out.

Ah, the dreaded transcription error. I've fallen afoul of it on countless occasions myself.

The point was that JREF were satisfied that 16/20 (the 80% of the claimant) was sufficient for this applicant to pass the Preliminary Challenge.

Yes, finding enough letters in time to put together a more rigourous test would have been tricky, I imagine.

I compiled a spreadsheet of the entire list (about 6 months out of date now) to determine just how many applicants get to a final protocol and the reasons for not getting there. It's actually difficult to analyse statistically because of the number of applications that aren't or don't need to be determined by odds compared to random chance. A number of them are "it either happens or it doesn't". [\quote]

Did anything come of your research? It sounds interesting.

Point I was trying to get across to the OP is that his fixation on JREF stating a standard "pass" rate is quite irrelevant and is a strawman constructed by him to discredit the Challenge.

It does seem that the challenge needs to be considered on a purely case by case basis. If only the applicants' abilities were so self evident as to make knowing the odds of passing by chance superfluous.