Is Randi's challenge scientific?

[plindboe]

It is my impression that Randi hasn't claimed that his tests are scientific, but that they still strive to make the tests somewhat scientific.

Perhaps what I'm wondering most of all is, what exactly is it that would make a test scientific? Just because a test uses specific scientific requirements, like double blinding, doesn't make it scientific. How much does it take, in order to refer to a test as scientific?

 

[Zep]

I think you may be mistaken in your original supposition.

The JREF Challenge seems to be designed to use fair and reasonable means, including quite possibly testing at a "professional science" level, in order to examine paranormal claims. As I see it, if a conclusive, fair and reasonable result that all will accept can be achieved using chewing-gum and an umbrella and a lemon, as opposed to hectares of bunsen burners and test-tubes and white lab-coats, then so be it.

For example, the testing of dowsers in Australia by the Australian Skeptics some years back used assorted water barrels, plastic pipes, taps and tonnes of dirt out in a field to construct a fair testing arena. Not a test-tube or lab-coat in sight. And yet it was scrupulously fair to all concerned, and did the job well.

The essence of the matter is that a testable claim must be made, and the testing method and expected results agreed to by both sides, BEFORE testing begins.

 

[plindboe]

The essence of the matter is that a testable claim must be made, and the testing method and expected results agreed to by both sides, BEFORE testing begins.

Indeed, I'm having a discussion with a proclaimed psychic and I must have explained that exact thing a billion times to her, but she keeps on repeating the same tiresome arguments.

For instance, she keeps saying Randi's challenge isn't scientific(though I believe she doesn't have a clue about what that means exactly, in even less degree than myself). So that's why I'm wondering, what exactly would constitute a scientific test, and is Randi's challenge scientific?

PS.I can give a link to the discussion in private, if anyone's interested, but I don't want to give it in the open, since I prefer to be the only skeptic there.

 

[Zep]

Ask her -

(a) What tests she would agree can prove beyond reasonable doubt that her psychic powers do or do not exist, and

(b) What it would take for her to agree she does NOT have any psychic powers.

I'm guessing the answers are (a) anecdotal evidence only, and (b) nothing will convince her.


I suppose you could also ask her what she thinks "scientific tests" actually means. Get her to describe it for you, either by concept or by example, so you both understand exactly what she means. (And I'm guessing you have already done this!)

 

[69dodge]

Originally posted by plindboe
For instance, she keeps saying Randi's challenge isn't scientific

Why does that bother her?

If she and Randi agree to a protocol for a test, and she passes the test, then she wins the money. Whether anyone considers it scientific doesn't really matter.

In order for her to agree to a protocol, it needs to be something she thinks she can do. In order for Randi to agree to a protocol, it needs to be something he thinks non-psychics can't do. Those are the only two requirements.

So, what can she do that non-psychics can't do?

 

[Luke T.]

There was an excellent Horizons television program last year in which Randi ponied up the million dollars in a homeopathy challenge. In this case, there were white lab coats and British scientists out the wazoo. Randi participated not at all, except to make a few predictions about how things would go.

It was a classic example of a double blind experiment and ended with the predicted results.

 

[Luke T.]

I would say that the accusation of Randi's experiment not being scientific is an attempt to say that since it isn't scientific, it leaves room for Randi to cheat. In other words, it doesn't leave any room for the challenger to cheat.

 

[Rolfe]

It depends on how you define "scientific". In my opinion, white coats and machines that go "beep" do not science make. Without looking it up, and entirely in my opinion, "science" describes meticulous, objective investigation designed to add to our knowledge of objective fact. However it is done.

In that case, the JREF Challenge is scientific.

Rolfe.

 

[Luke T.]

The Scientific Method

 

[wollery]

The challenge is a test designed to differentiate between two hypotheses.

1. The test subject has a prespecified paranormal ability.

2. They don't.

All parties agree before hand which results will support which hypothesis.

I don't think that there could be a better definition of a scientific test.

 

[drkitten]

Perhaps what I'm wondering most of all is, what exactly is it that would make a test scientific? Just because a test uses specific scientific requirements, like double blinding, doesn't make it scientific. How much does it take, in order to refer to a test as scientific?

The big thing that Randi's test lacks is an investigative purpose; science is mostly about investigation (finding stuff out), not about demonstration/challenging (showing stuff off). This isn't a criticism, by the way, but a description.

One of the complains that I've often heard (on this forum, no less) about Randi's challenge is that he's got too clear-cut, black/white, success/failure conditions. For example, if I claimed that I could make a roulette ball land on black 80% of the time (and Randi and I agreed on that threshhold), but in testing I could only manage 79/100 trials, Randi would say "You lose, feel free to reapply in a year." In "real" science, a "real" scientist would be really impressed by the near-miss and start trying to replicate this effect. Formally, to pass the preliminary test, you need a p value of 0.001 or less, for the final, of 0.000001 or less. If I scored a p value in testing 0.002, that's still "highly significant," and worthy of further research, but not acceptable to Randi

Randi doesn't do further research. There's three reasons for that --- first, he's not in the research business, but in the fraud-busting business. Second, there's a lot of money at stake and he has to have clear-cut rules for everyone to play by.

Third, by any sensible standard, if you're applying for the Challenge, you should already know what you can (and can't) do -- it's not a question of figuring something out, but of demonstrating a talent you claim to possess. There's no reason to "investigate" something you should already know. (This is part of the problem with Beth Clarkson's proposed TK protocol; she's a good statistician, but doesn't know whether or not the effect she claims really exists. She wants to do research on her ability [which I commend], but she should be able to do that in her own lab before approaching the JREF. Once she's found a reliable effect -- and she's a good mathematician, she knows what "reliable" means -- then talk to KRAMER.)

However, the tests are as clear-cut an example of how to run a scientific experiment as you could hope for. He does the experiment right, and better than a lot of card-carrying Ph.D-equipped professionals lab-nerds. He's one of the best experimental designers I know. However, he doesn't design experiments; he designs controlled demonstrations.

 

[pgwenthold]

Re: Re: Is Randi's challenge scientific?

One of the complains that I've often heard (on this forum, no less) about Randi's challenge is that he's got too clear-cut, black/white, success/failure conditions. For example, if I claimed that I could make a roulette ball land on black 80% of the time (and Randi and I agreed on that threshhold), but in testing I could only manage 79/100 trials, Randi would say "You lose, feel free to reapply in a year."

The thing that always gets overlooked is that the claimnant is perfectly able to control this. If you think you can hit 80%, tell Randi that you get at least 150/200. Determine beforehand what is the lower limit of what is acceptable. If you think it is 80%, know the standard deviation for the number of trials you are going to do and chose a limit 4 SD's below 80%. That means that if you actually can get it right 80% of the time, as you claim, then you have more than a 99% chance of winning.

In 100 trials, that is a SD of about 3.5, so if you say you can get 68 right, you will be safe.

Now, Randi may not accept 68% in 100 trials, so go to more trials. Here's about how many you need given the number of trials

100: 68 (68%)
250: 175 (70%)
500: 366 (73%)
1000: 753 (75%)

I would guess that if you told Randi you could get 75% out of 1000 correct in a 50/50 choice, he'd accept it. If you actually can get it right 80% of the time, then 750/1000 is pretty much guaranteed. I just ran 1000 samples of random trials and the lowest I got was 761/1000. I got 753 once in a second thousand, so close, but still over the 750 threshold. Indeed, with 10000 trials, the lowest I got was 753. Therefore, if you are actually able to predict red/black 80% of the time, just tell Randi you will get 750 out of 1000. You are pretty much guaranteed to win.

Assuming, of course, you can actually get it right 80% of the time.

 

[drkitten]

Re: Re: Re: Is Randi's challenge scientific?

The thing that always gets overlooked is that the claimnant is perfectly able to control this. If you think you can hit 80%, tell Randi that you get at least 150/200.

Not disagreeing with what you say --- but this also points up the fundamental difference between the Randi Challenge and "science."

Science is concerned with the question "is there (possibly) an effect?" Randi is concerned with the question "has the claimant unequivocably demonstrated an effect?"

Because of this different, Randi and "science" handle ambiguous results differently, and even the analysis procedure is a little bit different (a scientist would gather data first, then run the stats to figure out if the data meant anything. Your proposal above reverses that order).

 

[Luke T.]

One of my biggest pet peeves about the paranormal community is the way they hijack the hard work of real scientists and twist it into gibberish and consider the result just as scientific.

It is my hypothesis ( ) that because the paranormal community has mired itself so deeply in this gibberish which they believe to be "science", they haven't a clue what the hell science is, much less a scientific experiment. Not when you have psychics who bloviate impressive mathematical claims of "90 percent hit rates" and crap like that.

So when Randi tries over and over to explain that in order to be tested you FIRST have to explain what the hell it is you can do that is so freaking special, he is often already speaking over their heads. Never mind getting into the nuts and bolts of how your ability will actually be tested. That's just too far into the deep end for these whackos to understand.

Thus we get freaks like Lucianarchy who thinks he won the million bucks because he posted "ladybrook" in a post on this forum and believes, or wants others to believe, he was robbed.

This kind of unstructured thinking leads to the argument that someone claiming a paranormal ability "proved" their ability, Randi didn't pay up, which means Randi cheats, which means the test is unscientific.

Totally, utterly, bogus.

 

[pgwenthold]

One of my biggest pet peeves about the paranormal community is the way they hijack the hard work of real scientists and twist it into gibberish and consider the result just as scientific.

It is my hypothesis ( ) that because the paranormal community has mired itself so deeply in this gibberish which they believe to be "science", they haven't a clue what the hell science is, much less a scientific experiment. Not when you have psychics who bloviate impressive mathematical claims of "90 percent hit rates" and crap like that.

So when Randi tries over and over to explain that in order to be tested you FIRST have to explain what the hell it is you can do that is so freaking special, he is often already speaking over their heads. Never mind getting into the nuts and bolts of how your ability will actually be tested. That's just too far into the deep end for these whackos to understand.

This is the big thing. I don't know that Randi ever really says he is doing scientific testing of psychics. It is always, "We are testing to see if you can do what you claim to be able to do."

And as you note, the biggest problem is getting them to come out and say exactly what it is they can do. Ask a psychic what they can do, they say, "I'm psychic." What does that mean? "I can tell things by using my psychic abilities."

Keep going, and going, but you never get anything specific for an answer. Of course, it is important that they never actually define their abilities, because in that case, they know that they can be shown to be wrong, so they keep it as vague as possible to leave it non-testable.

Look at the pulling of teeth that Randi went through with Sylvia Browne. It took forever to come up with a protocol, because she kept asking, "How many do I have to get right?" and he kept saying, "It depends on what you claim you are able to do." She never caught on that SHE was the one who needed to tell HIM what she was able to do. If she is able to correctly see 75% of the information, then he would devise a test that would require her to demonstrate a 75% success rate. Apparently, she wanted him to provide the minimum requirements for "passing," but that isn't what it is about. It is about their claims. Randi never says, "I bet you can't get XX," he says, "You have claimed to be psychic. I'll give you a million dollars if you can do something you claim to do by psychic means."

 

[plindboe]

Thanks for the great replies!

The discussion I have with this woman is the most frustrating discussion I have ever had. She has the intellectual capability of a child, and I have at great length tried to explain stuff to her, only to see the same refuted arguments resurface in the very next post. Her husband is a scientist, and has helped her with her posts and main arguments, so she just keeps repeating them ad nauseam, even though I have put each and every one of her points to shame.

The one she repeats the most is that since there are different standards in each case it's fraud and manipulation. I have explained over and over that it is necessary, because applicants have unique requirements, and yet she repeats it with the very same conviction as before.

I wonder, whether I have died a week ago, and ended up in Hell. (Perhaps that could win the challenge)

 

[patnray]

The one she repeats the most is that since there are different standards in each case it's fraud and manipulation. I have explained over and over that it is necessary, because applicants have unique requirements, and yet she repeats it with the very same conviction as before.

Have you tried asking her what ONE standard she would use to evaluate the following 3 claimants?
1) Claims to be able to detect buried water pipes by dousing with 100% accuaracy.
2) Claims to be able to diagnose medical conditions by looking at a person. Doesn't claim to detect every condition but claims she is never wrong about a condition she does detect.
3) Claims to be able to predict whether a roulette ball will land on red or black 60% of the time.

Then ask her to pick just one number for the number of trials that would be needed to test all three claims and one number for the success rate that would be considered "passing" for all three claims.

 

[hgc]

Have you tried asking her what ONE standard she would use to evaluate the following 3 claimants?
1) Claims to be able to detect buried water pipes by dousing with 100% accuaracy.
2) Claims to be able to diagnose medical conditions by looking at a person. Doesn't claim to detect every condition but claims she is never wrong about a condition she does detect.
3) Claims to be able to predict whether a roulette ball will land on red or black 60% of the time.

Then ask her to pick just one number for the number of trials that would be needed to test all three claims and one number for the success rate that would be considered "passing" for all three claims.

Don't do it. She'll blow a gasket.

 

[CBL4]

I do not think the level of accuracy (e.g. 60% or 80%) is that important. I presume that Randi just wants to get a p level so low that he will never pay out by chance.

Assume you and Randi agree to a level of 80 out of 100 because it provides a p < .00001 (or whatever). If you get a score of 75 (p < .0001 or whatever), then Randi keeps his money but you also get to say Randi has proven your psychic abilities. Even though you do not get $1,000,000 from Randi, you can probably get a lot more from other people.

CBL

 

[pgwenthold]

I do not think the level of accuracy (e.g. 60% or 80%) is that important. I presume that Randi just wants to get a p level so low that he will never pay out by chance.

Assume you and Randi agree to a level of 80 out of 100 because it provides a p < .00001 (or whatever). If you get a score of 75 (p < .0001 or whatever), then Randi keeps his money but you also get to say Randi has proven your psychic abilities. Even though you do not get $1,000,000 from Randi, you can probably get a lot more from other people.

CBL

It can't work this way. The number of trials needed is based on the claimed ability, not the other way around. This is done to protect the claimnant, in fact. If claim I can predict 55% of coin flips, it is not fair to require me to get 80 out of 100. However, if I can actually predict 55% of coin flips, then I most certainly do have a non-standard ability and am certainly eligible for the million dollars. By your standards, I pretty much could not win despite having paranormal ability. THAT would be unfair.

Instead, tests are designed for me to show what I claim to be able to do. If I claim I am able to get 55%, I have to be able to do it to a level where p < 0.000001. You can't do that in 100 trials, or even 1000, for that matter. It would take a very, very large number to trials to get 55% to be significant. But that is what I would have to do to win the money. On the other hand, if I have the ability, then I am not concerned, given my built in safeguards (see above about 4 SD)

The short answer: 80% in 100 trials is far too stringent if I claim I can only get 60% right. In that case, we need to go to more trials. It all depends on what I claim I can do.