Merged Odds Standard for Preliminary Test

[drkitten]

No one's stopping claimants, or anyone else for that matter, from being tested under proper observing conditions (ie, where all opportunity to cheat is eliminated), yet none of the claimants attempting the MDC have done so.

This is, not to put too fine a point on it, a stupid comment.

You're asking that claimants attempting the MDC should have passed the MDC already. Part of the issue is that "proper observing conditions (ie. where all opportunity to cheat is eliminated)" is one of the specialist skills that Randi and company provide. One of the things he has said (repeatedly) is that scientists should consult magicians when studying the paranormal. Well, Randi is a highly skilled magician and available more or less for free --- why should I pay money for a prior consultation with a less skilled magician who doesn't specialize in detecting cheating?

Setting up "proper observing conditions" is hard work. Why should the claimants have to do all that hard work by themselves, without benefit of Randi's knowledge and expertise, in order to avail themselves of it later?

Oh, and a possible paranormal ability that can't be easily tested within a day? "I can make plants grow faster/taller/bushier." (Which is a genuine claim that people have made.) At a minimum, this will require several weeks of work and observation; if the plants in question are redwood trees, it could take several hundred years. Pyramid power --- "if you store razor blades in this, they will stay sharp for longer" --- is another classic woo claim that would be be hard to test; an unused razor blade will stay sharp for months or years, depending upon the climate. Sure, it would be paranormal if one stayed sharp for two years while the other only stayed sharp for six months ---- but you're not going to notice a difference after a day or a week.

If the JREF uses a day as their threshold, even simple claims like "drinking this magic water will make you resistant to disease" are untestable.

 

[drkitten]

Of course, but experts can still be hired, and if a test can only be done by properly paid experts, this should be made clear to the claimant, and if s/he cannot carry the costs, the claim has to be rejected because of this. It seems wrong to force the claimant to rely on chance in order to keep down costs.

And that's part of the challenge negotiations, isn't it?

Part of the issue is precisely that claimants may not know what the "requirements" for testing are, and part of the "Educational" part of JREF is teaching them about proper scientific investigation, including the role of experts. The JREF, in a very strong sense, is doing the claimants a favor by keeping a Rolodex full of experts who are willing to work for free to help test claims.

Heck, the JREF is doing them a favor simply by vetting the experts to keep them from pouring money into consulting Doctor Marvel the Snake Oil Salesman with a mail-order Ph.D.

Having said this, though,.... we still have the negotiations part of the negotiations. If the JREF will not run a test longer than 8 hours (which is not an unreasonable general policy), and the claimant cannot establish his claim within 8 hours, then there is a genuine impasse and the test should not be run. But is that really a best outcome? Or is a better outcome for the claimant to voluntarily accept a (free) test that he may or may not be able to pass?

 

[Moochie]

Yeah.

Another way of looking at it is that if you're so messed up that you can't convince two academics and a journalist of your abilities (given how gullible academics and journalists are known to be), the JREF is not the place for you.

The problem is (and you can check the records on-line) that most of the claimants WERE sufficiently messed up that it seemed that the JREF was running an outpatient clinic, not an educational foundation.

Well, yeah.


M.

 

[Moochie]

This is, not to put too fine a point on it, a stupid comment.

Gee, thanks.

You're asking that claimants attempting the MDC should have passed the MDC already. Part of the issue is that "proper observing conditions (ie. where all opportunity to cheat is eliminated)" is one of the specialist skills that Randi and company provide. One of the things he has said (repeatedly) is that scientists should consult magicians when studying the paranormal. Well, Randi is a highly skilled magician and available more or less for free --- why should I pay money for a prior consultation with a less skilled magician who doesn't specialize in detecting cheating?

OK, maybe I chose my words unwisely. What I would think is that people who have convinced themselves they have paranormal abilities would be shouting it from the rooftops; ie, we surely would have heard something by now. It wouldn't have to be tests done to a scientific certainty, but tests conducted publicly with some media present. The revised rule about "media presence" goes some way toward this, but not far enough, in my opinion.

Setting up "proper observing conditions" is hard work. Why should the claimants have to do all that hard work by themselves, without benefit of Randi's knowledge and expertise, in order to avail themselves of it later?

If there was anything to their claims, they would have capitalized on them by now, I would think, and not necessarily in a monetary sense.

Oh, and a possible paranormal ability that can't be easily tested within a day? "I can make plants grow faster/taller/bushier." (Which is a genuine claim that people have made.) At a minimum, this will require several weeks of work and observation; if the plants in question are redwood trees, it could take several hundred years. Pyramid power --- "if you store razor blades in this, they will stay sharp for longer" --- is another classic woo claim that would be be hard to test; an unused razor blade will stay sharp for months or years, depending upon the climate. Sure, it would be paranormal if one stayed sharp for two years while the other only stayed sharp for six months ---- but you're not going to notice a difference after a day or a week.

If the JREF uses a day as their threshold, even simple claims like "drinking this magic water will make you resistant to disease" are untestable.

Now you appear to be shilling for woodom. Good luck.


M.

 

[steenkh]

And that's part of the challenge negotiations, isn't it?

My point was that it does not seem to be part of challenge negotiations.

With Pavel we have a claimant who seems to be serious about testing himself, and who claims a success rate significant better than chance - but who is forced to accept a test where he can only succeed if he is even better than his usual performance. I am not saying that it is wrong of the JREF to insist on a shorter and cheaper test, but the possibility of making a longer, expensive test has not been part of the negotiations, as far as we know.

 

[Rodney]

My point was that it does not seem to be part of challenge negotiations.

With Pavel we have a claimant who seems to be serious about testing himself, and who claims a success rate significant better than chance - but who is forced to accept a test where he can only succeed if he is even better than his usual performance. I am not saying that it is wrong of the JREF to insist on a shorter and cheaper test, but the possibility of making a longer, expensive test has not been part of the negotiations, as far as we know.

Again, a way around this dilemma is to permit a continuation of the preliminary test when the results are well above chance, but insufficient to pass that test after just 40 trials. If Pavel has no ability to do what he claims, his number of hits should be no higher than 24 out of 40. If he does better than that, either he has the claimed ability or he got lucky. The notion that, even if he gets 29 correct, "he failed the preliminary test" is illogical, when that number of hits would be better than his claimed average.

 

[Garrette]

Again, a way around this dilemma is to permit a continuation of the preliminary test when the results are well above chance, but insufficient to pass that test after just 40 trials. If Pavel has no ability to do what he claims, his number of hits should be no higher than 24 out of 40. If he does better than that, either he has the claimed ability or he got lucky. The notion that, even if he gets 29 correct, "he failed the preliminary test" is illogical, when that number of hits would be better than his claimed average.

Non-statistician that I am, this makes more sense than your previous posts on the topic. Tell me if I am correctly re-stating it:

Claimant has X hours to do Y.

A% or greater success means immediate passing of the preliminary test

B% or greater (but less than A%) means the test has not been passed but will continue for Z more hours, after which the two tests are combined to determine success or failure with no possibility of another extension.

Less than B% means failure of the preliminary test.


Though this may make some statistical sense, it does not address the problem of extended tests. The experts, volunteers, et all must be prepared to stay longer than the original X hours which is the same thing as requiring them to do so.

 

[Cuddles]

Though this may make some statistical sense, it does not address the problem of extended tests. The experts, volunteers, et all must be prepared to stay longer than the original X hours which is the same thing as requiring them to do so.

It also doesn't address the problem that it's nothing to do with the challenge. The challenge is very simple. Applicant states they can do x. Applicant either does x or fails to do x. End of story. As Rodney consistently resfuses to acknowledge, the JREF is not a scientific research institute. Someone failing to do what they claim but still getting results that some people find interesting could well be a valid topic for study. And the failed applicant, and anyone else who feels like it, are free to go and do so. However, the JREF will not do so, and has no responsibility to do so.

 

[Garrette]

It also doesn't address the problem that it's nothing to do with the challenge. The challenge is very simple. Applicant states they can do x. Applicant either does x or fails to do x. End of story. As Rodney consistently resfuses to acknowledge, the JREF is not a scientific research institute. Someone failing to do what they claim but still getting results that some people find interesting could well be a valid topic for study. And the failed applicant, and anyone else who feels like it, are free to go and do so. However, the JREF will not do so, and has no responsibility to do so.

I agree with you, but I was addressing it as if the intent were to offer a solution to the problem of too-lengthy testing as opposed to the intent being one to turn the MDC into something it is not.

I run into this a fair amount, not only here but in real life and on other forums.

 

[Pope130]

I can find nowhere that Pavel is being forced to accept a test. A protocol has been suggested based on his claimed abilities. If he finds it unacceptable he should propose a different protocol or changes to the existing one that would work.
If he feels the need for a longer test, say one that would take two days to complete, he should propose that. Then see what JREF says. Over in the Homeoproofer thread they are discussing a protocol that will take weeks or months to complete.
It is for the challenger to make the claim. State what he can do and how it will be tested, what constitutes a success or failure. Claiming that the JREF is forcing a failure is just pre-positioning an excuse for failure.

 

[Startz]

It also doesn't address the problem that it's nothing to do with the challenge. The challenge is very simple. Applicant states they can do x. Applicant either does x or fails to do x.

I can find nowhere that Pavel is being forced to accept a test. A protocol has been suggested based on his claimed abilities. If he finds it unacceptable he should propose a different protocol or changes to the existing one that would work.

With all due respect, perhaps there is some misunderstanding here about how the Challenge actually operates. Both JREF and the applicant have to agree to the terms of the protocol. It is not just whatever the applicant states.

Pavel originally offered a different protocol, which JREF declined. While the test itself requires "no judgement," this isn't true of the protocol design. Pavel has agreed to changes in order to move the process along. Reasonable people can disagree over whether JREF's or Pavel's position is better.

Mind you, JREF has not yet offered an opinion on the current protocol. I would hope they would permit a somewhat longer test in order to afford Pavel a better opportunity to demonstrate he can do what he has claimed he can do.

 

[Rodney]

Non-statistician that I am, this makes more sense than your previous posts on the topic. Tell me if I am correctly re-stating it:

Claimant has X hours to do Y.

A% or greater success means immediate passing of the preliminary test

B% or greater (but less than A%) means the test has not been passed but will continue for Z more hours, after which the two tests are combined to determine success or failure with no possibility of another extension.

Less than B% means failure of the preliminary test.

Correct, except that there could be another extension if the results are even further above chance than they were after the first test, but still insufficient to pass the preliminary test.

Though this may make some statistical sense, it does not address the problem of extended tests. The experts, volunteers, et all must be prepared to stay longer than the original X hours which is the same thing as requiring them to do so.

I'm not necessarily suggesting that the preliminary test be continued at that moment, but rather at least well in advance of the usual one-year period for taking another preliminary test. That's particularly important in view of the fact that the MDC is scheduled to end in March 2010. Let's suppose that in September 2008 Pavel "fails" the test with only 29 hits, and then has to wait until September 2009 for the re-test. Suppose he "fails" again with 29 hits. Does the JREF say: "Sorry, you can't take another preliminary test, but please join us in March 2010 for our celebration of the fact that we never had to cough up the million bucks"?

 

[Pope130]

Pavel has agreed to changes in order to move the process along.

My point was that (at this point) this is a negotiation. Pavel is now complaining that he agreed to a test he believes he can not pass. He has not been forced into this agreement. He should make a counter proposal. Perhaps there is a point where the protocol will be acceptable to JREF, and Pavel feels he can pass.

 

[petre]

Correct, except that there could be another extension if the results are even further above chance than they were after the first test, but still insufficient to pass the preliminary test.


I'm not necessarily suggesting that the preliminary test be continued at that moment, but rather at least well in advance of the usual one-year period for taking another preliminary test. That's particularly important in view of the fact that the MDC is scheduled to end in March 2010. Let's suppose that in September 2008 Pavel "fails" the test with only 29 hits, and then has to wait until September 2009 for the re-test. Suppose he "fails" again with 29 hits. Does the JREF say: "Sorry, you can't take another preliminary test, but please join us in March 2010 for our celebration of the fact that we never had to cough up the million bucks"?

First just a quick shout to Moochie for the earlier question, when I said "there exist possible paranormal...blah blah blah...that could not be tested in a reasonable time frame", it may have been better phrased "one could describe something clearly paranormal that could not be tested in a reasonable time frame". In other words, just because a claim indicates something clearly paranormal doesn't mean it is automatically eligable for the challenge.

Second, to answer Rodney's question, I see no rule at all against an applicant asking for a multiple-trial test where a certain threshold performance would not be considered a pass of the preliminary, and yet would make the applicant eligable for a retest at some near-future time. I imagine the JREF is likely to be reluctant to set up multiple trial dates, since it would tend to indicate an ability so minor or so unpracticed that it is difficult to even discern it from natural fluctuations in probability. I expect the JREF would work with the applicant to find if there were any way to better test the ability within a shorter time frame.

 

[dbw]

Say a claimant says he has an (admittedly weak) ability to make a fair coin unfair... specifically, he claims he can meditate over a coin, without touching it, for five minutes, and that from then on, that coin will flip heads 50.0001% of the time. For the sake of argument, assume this guy really can do what he says. This is clearly a paranormal ability, but I submit that there's no way to test it in a reasonable amount of time. Thus the JREF would deem it untestable, though this is not strictly true.

...I think this is what petre was getting at. If not, it's still a valid point. And it's not really that farfetched... remember the woman who said she could influence a candle's flame "intermittently"? I think she was deceiving herself, but unfortunately we'll never know for sure as it would have taken forever to test.

 

[steenkh]

Again, a way around this dilemma is to permit a continuation of the preliminary test when the results are well above chance, but insufficient to pass that test after just 40 trials. If Pavel has no ability to do what he claims, his number of hits should be no higher than 24 out of 40. If he does better than that, either he has the claimed ability or he got lucky. The notion that, even if he gets 29 correct, "he failed the preliminary test" is illogical, when that number of hits would be better than his claimed average.

He claims he can do 30, and then 30 is the number that he needs to do, not 29. This is a challenge for the claimants to do as they claim they can do, not a scientific test to establish if the paranormal exists.

If a re-test was acceptable when achieving near-misses, the challenge protocol would have to be reworked to be much more complicated, if the same low level of false positives is to be maintained. What about Pavel simply claiming what he can actually do, consistently, rather than accepting odds he most likely cannot achieve?

Perhaps the problem lies in the fact that Pavel had not worked out the odds before he started, but is only now testing his ability and finding that he cannot consistently do what he has agreed to. Tough. But then I would suggest that he uses the next year to prepare for the last testing before the challenge closes down.

 

[fls]

Non-statistician that I am, this makes more sense than your previous posts on the topic. Tell me if I am correctly re-stating it:

Claimant has X hours to do Y.

A% or greater success means immediate passing of the preliminary test

B% or greater (but less than A%) means the test has not been passed but will continue for Z more hours, after which the two tests are combined to determine success or failure with no possibility of another extension.

Less than B% means failure of the preliminary test.

Though this may make some statistical sense...

It doesn't. What you have just described is 'early stopping', a well-known* way of introducing bias and increasing the possibility of obtaining a positive result.

Linda

*To be defined to my advantage at a later date.

 

[Garrette]

It doesn't. What you have just described is 'early stopping', a well-known* way of introducing bias and increasing the possibility of obtaining a positive result.

Linda

*To be defined to my advantage at a later date.

But, but, but....

I'm an intelligent layman! I HAVE to be right!

 

[Garrette]

Correct, except that there could be another extension if the results are even further above chance than they were after the first test, but still insufficient to pass the preliminary test.


I'm not necessarily suggesting that the preliminary test be continued at that moment, but rather at least well in advance of the usual one-year period for taking another preliminary test. That's particularly important in view of the fact that the MDC is scheduled to end in March 2010. Let's suppose that in September 2008 Pavel "fails" the test with only 29 hits, and then has to wait until September 2009 for the re-test. Suppose he "fails" again with 29 hits. Does the JREF say: "Sorry, you can't take another preliminary test, but please join us in March 2010 for our celebration of the fact that we never had to cough up the million bucks"?

See Linda's response above. Based on that, I retract my comment about it making statistical sense.

 

[Garrette]

With all due respect, perhaps there is some misunderstanding here about how the Challenge actually operates. Both JREF and the applicant have to agree to the terms of the protocol. It is not just whatever the applicant states.

Pavel originally offered a different protocol, which JREF declined. While the test itself requires "no judgement," this isn't true of the protocol design. Pavel has agreed to changes in order to move the process along. Reasonable people can disagree over whether JREF's or Pavel's position is better.

Mind you, JREF has not yet offered an opinion on the current protocol. I would hope they would permit a somewhat longer test in order to afford Pavel a better opportunity to demonstrate he can do what he has claimed he can do.

I don't think Cuddles et al are doing what you say here.

One must distinguish between the claim and the protocol to test that claim. The claim is what the applicant, and only the applicant, can state. The protocol is what requires negotiation and agreement.