Guidelines For Testing

[H3LL]

Do we have any helpful guidelines for testing claims?

We have the rules, SWIFT and commentaries to sift through but a forum or web page with broad guidelines for certain claims would be useful.

Not only for any skeptics to learn from but also people with potential claims. It is always suggested that claimants double blind test themselves before submitting a claim, but we are a bit short on any guidelines (or have I missed something).

If we don't have them in one place we could put suggestions here and examine them for holes.

Once a suggestion is well examined we can move on to another and so-on.

I do stress that this would only be guidelines for claimants to check themselves and not in anyway a hard and fast protocol for testing as all claims are individual.

I would be happy to record the details as they evolve and prepare a document for final clearing by the JREF team and we could maybe have it as a sticky in a dedicated thread.

What are your thoughts?

<hr>

Suggested self-testing procedures:

Webpage of Work-In-Progress (THIS IS A DEAD LINK FOR NOW)

Download Document of Work-In-Progress (THIS IS A DEAD LINK FOR NOW)

<hr>

 

[Dr Adequate]

Some points to bear in mind when designing an experiment.

(1) Everyone is biased and no-one is objective. The experiment must be designed so that the preconceptions of the researcher can't affect the outcome. A psi test, for example, should be so designed that it doesn't matter whether the researcher is a believer or sceptic (see the notorious "sheep and goats effect").

(2) Make sure that you're measuring what you think you're measuring. If you are claiming evidence of one effect, make very sure that it's not caused by some other effect. Ask around to see if anyone can propose another explanation. See if you can design the experiment so as to eliminate that explanation. Callibrate your instruments. If you find a magic effect involving Magic Crystals, see if Non-Magic Glass has the same effect. Do this double-blind (see point 1).

(3) What constitutes good experimental design is not obvious. If you think it is, and that you've got it sussed, you are so wrong. Learn from the past. The history of science is full of good examples (Clever Hans) and Awful Warnings (the N-Rays fiasco) concerning the design of experiments.

(4) Ask other people, especially scientists, if you have a scientist available, if they think the experimental design is OK before you take the trouble of doing the experiment. If it turns afterwards out that the method was flawed, then you've just wasted your time, no matyter how appealing the results were.

 

[Paul C. Anagnostopoulos]

Ask a magician to review the protocol. He would probably be good at exposing sensory leaks even without attending the actual experiments.

Never forget the clock synchronization leak in Sheldrake's telephone telepathy experiments.

~~ Paul

 

[Ashles]

(2) Make sure that you're measuring what you think you're measuring. If you are claiming evidence of one effect, make very sure that it's not caused by some other effect. Ask around to see if anyone can propose another explanation. See if you can design the experiment so as to eliminate that explanation. Callibrate your instruments. If you find a magic effect involving Magic Crystals, see if Non-Magic Glass has the same effect. Do this double-blind (see point 1).

For claimants who genuinely believe in their ability I think this is the most important, and also, probably the hardest for them to do.

They must try and put themselves in the place of a sceptical observer and think "What possible other mechanisms could produce this effect, and can I eliminate those possibilities before I am tested"

In short they must test themselves several times, changing the protocol in as many key ways as possible but still displaying the claimed result.

Pragmatist has an excellent example here that is worth a read by any potential claimant.

 

[H3LL]

Thank you.

They're a great starter. They would be nice as a general summary of testing proceedures. Any critical observations?

I was thinking that once we have that, we could move on to specific claims such as a Seft test for dowsing etc.

Let's see how it develops.

 

[JMA]

There is a good book on that subject by Wisemand & Morris :

"How to test psychic claimants?"

You can find it easily on the web...

It's a lot of guidelines in order to test psychic. It's general for every type of psychic (medium, ESP, PK, and so on) so you'll have to come here, on top of that book, and read the "Million Dollar Challenge" topics for more specific advices... It's what I do and it's really usefull...

 

[Carn]

(3) What constitutes good experimental design is not obvious. If you think it is, and that you've got it sussed, you are so wrong. Learn from the past. The history of science is full of good examples (Clever Hans) and Awful Warnings (the N-Rays fiasco) concerning the design of experiments.

So i'm wrong, because i think a good experimental design is obvious. Care to post some example, where the good experiment is not obvious?

Carn

 

[Dr Adequate]

So i'm wrong, because i think a good experimental design is obvious. Care to post some example, where the good experiment is not obvious?

You will see there's still quite a lot of discussion that goes on in the Million Dollar Challenge thread about what is and isn't a good protocol to test a new claim. It's not "obvious" even to KRAMER, who's an old hand. Maybe you could give him a few tips now and then. But what you, or even I, find obvious and what others find obvious may not be the same. How many people on that healthyforums site do you suppose could find the flaw in a flawed experimental design? How well would a sceptic do, with no particular knowledge of science or the history of science (thaiboxerken might be an example)? This is why I put learning from the past among the foundations of what an investigator needs to do.

So let's look at the past, and see if good experimental design is "obvious". Look how long it took to find a good protocol for testing Clever Hans. Look at the flaws in the early psi experiments, which Randi et al exploited to such comic effect. Look at the early tests of Geller. Look at the Superminds fiasco --- metal-bending children. (The perfect experiment turned out to involve a one-way mirror, and a deliberate relaxation of vigilance on the part of the ostensible observers.) Look at the N-ray fiasco. Can't people tell when something is brighter or darker? Not when wishful thinking comes into play. Now these are examples involving real scientists, not just amateur believers or sceptics: and the good experimental design wasn't immediately obvious to them. Looking back we can learn from their mistakes, see what they've done --- their mistakes may be "obvious" to us, now, but only because we've taken the trouble to learn from their mistakes --- as I advise.

"Obvious", you say? Epimetheus had it easy.

 

[Ashles]

Carn, look at something simple like dowsing.

You'd think you should just get 10 bottles of water, paint them black then fill some with water then get the dowser to test.

That would be an obvious test.

But it wouldn't be a particularly good test.

Because you know where the water-filled bottles are you might unconsciously 'cue' the tester when they approached those bottles and give it away. So you should employ the double-blind protocol by which a third party chooses which bottles are placed where, so the experimenter and the dowser bioth have no idea which bottles are which until the end of the experiment.

Then, have you tested for the presence of underground streams? If dowsing works then maybe the dowser might pick up underground streams and this would interfere with their test. They could get a negative result, despite having actually detected water - just the wrong water.
Best to have the dowser check the area first (without bottles) to be sure.

Also, get the dowser to perform the experiment with clear bottles (water easily visible) first so they can confirm no other influences are at work.

Now you are closer to having a good protocol.

When it comes to tests of the paranormal the 'obvious' protocol rarely takes into account all the possible methods that the claimant could conceivably use.

I like to think I can see trickery a good part of the time, but if I had been testing that little girl (Natalia Lulova ) I would have been stumped as to how she could still read when wearing what appeared to be a sturdy blindfold.
The 'obvious' experimental procedure would have failed to detect her method.

An 'obvious' experimental protocol would work for claims like "I can fly" or "I can make myself magnetic".
But, curiously, the JREF never gets straightforward claims like that.

 

[Dr Adequate]

Also, get the dowser to perform the experiment with clear bottles (water easily visible) first so they can confirm no other influences are at work.

Except that then painted/unpainted becomes a variable that might be claimed to influence the test.

 

[Ashles]

Except that then painted/unpainted becomes a variable that might be claimed to influence the test.

I know, but I was getting bored by that point and couldn't be bothered to include: - get the dowser to dowse the painted bottled when empty, then fill some of the bottles with water and let them know which they are then get them to agree the paint hasn't affected the dowsing.

Anyway it was obvious.

 

[Carn]

After thinking about it, i realize, that i think, that i think its obvious, which first steps lead in the direction of good experiments, while of course deciding, if the design is already good enough is difficult.

Therefore i have big problems to understand, why there are claimants, who are suprised/angered by the test protocols and changes JREF asks for.

E.g. the russian girl trying to read while blind folded and Randi tries to make sure the blind fold realy fits and Mr.Palant does not even seem to grip, why this is sensible to check and correct.
Or dowsers, who never had the idea, to see if their ability works, even if there is no way for them to give good guesses from experience.

I mean the first question i asked(first time i ever heard/read a para claim), when some friends told me, they score very good, when taking a pack of cards of five different colours and mind reading, what colour the other draw, was "how do you make sure, that your stance and voice(they did it siting in the same room), do not give away, what colour you have". And i was confused to learn, that they never thought about that problem and tried to solve it.

I have a idea for further advice, though they might be in some of the above and have to reworded to take any offense away:

- Do not trust your instincts, feelings and hopes. Consider yourself to be deluded and recheck every step that led you to be convinced you have something paranormal for signs of delusion.

-Remember chance and coincidence(if you do not think they exists, just assume they exist)


Carn