• Quick note - the problem with Youtube videos not embedding on the forum appears to have been fixed, thanks to ZiprHead. If you do still see problems let me know.

Check my methodology - prayer study

DAO - Absolutely not. If you read the Recipients page, at the bottom it explicitly says that we recommend they continue their normal treatment.

Recipients report purely through questionaire, most of which will consist of t/f, numeric, or quantiative-scale measures. There will be opportunity for them to give anecdotal text responses as well, but those are not used for the final analysis.
 
Update: Exerpted quote from email w/ Jeff:

Yes, if you could work out a double-blinded protocol, it could work.
Jeff Wagg

On 8/23/06, Sai wrote:
[...]
However, I would like you to answer your implied assertion directly:
if we can work out a double-blind protocol, would this type of
challenge be acceptable?
Click to expand...

So it looks like we're on, at least in theory.
 
DAO: Simple: the difference between active and control groups of recipients.

Active group gets prayed for by our people, control doesn't. They are otherwise purely random assignment, and they don't know what their status is.

So if prayer works (or more specifically, if it makes a difference whether we assign people to pray for them or not), then there should be a difference between those two groups in terms of their outcomes (e.g. pain, $ spent, whatever).
 
Thanks again, Sai,
I must say this is one of the more interesting proposals I've seen around here in a long time.
Theoretically, after the first year, isn't it possible that a prayed-for recipient's condition could actually be worse than it was at the beginning of the experiment, yet they report that they feel better than ever? Wouldn't an actual physical examination by a medical team be far more informative than the perceptions of the recipient?
 
DAO - That is the purpose of having their physician fill out a short survey about the patient once per quarter. ;)
 
P.S. If they report that they feel better, that still counts as an effect. Pain relief, y'know.
 
Sai,

So far, you seem genuinely forthright. I detect no B.S. It does not sound like you are trying to snag Randi's million through fraud. I'm a skeptic to my core. I fully anticipate that if this project is nurtured to full fruition, it will clearly show that prayer has absolutely no effect. Nevertheless, your proposal is interesting.

The other posters in this thread will offer all kinds of advice on your protocol. Please give all advice due consideration. I'm sure it will be offered in the spirit of conducting a proper, double-blind test. If you fail to heed their suggestions and demonstrate you are unwilling to close loopholes within the protocol, my B.S. meter will begin to rise and I'll urge that further negotiations cease. But, so far, so good. Good luck.
 
DAO - Thanks.

I am indeed quite earnest. And, evidently unlike some portion of applicants, I'm sentient too. ;) It may surprise you to know that I'm also a skeptic.

FWIW you may consider me completely agnostic as to whether the test will be successful or not; I simply want to conduct it. While I'm at it I may as well get JREF MDC involved, so nobody can say that my methodology was flawed or I was faking it or anything (as I have no intention or doing so). Either outcome is quite acceptable to me; my religious/spirtiual beliefs are in no way harmed either way.

I encourage anyone who can point out a genuine flaw in the protocol to point it out, but I do need to require that they be able to explain how their perceived flaw would cause a false positive. I contend that the methodology is flawless (i.e. that it will not cause a false positive; I am willing to accept false negatives); prove me wrong. :)
 
saizai said:
Digithead - What Startz said.

Or to quote myself in the OP:
"If you have a critique, please make sure that:
* you can explain why your perceived flaw in my design would cause a false positive result, i.e. a statistically significant difference between the active and control groups of Recipients in the second and/or third round"

Can you give that explanation?
Click to expand...


You can't just assume that your randomization scheme worked, you have to demonstrate that it worked. You also need to check this after you assign them to their respective groups but before you perform the treatment.You'd be suprised how often in clinical trials the randomization schemes didn't work out as intended...

And you can get a false positive by loading up one of the groups with people who had a less severe form of cancer who thereby benefitted most from their medical treatment rather than prayer but it looks like it was prayer that did it. Hence why you have to risk-adjust in your statistical tests...
 
Startz said:
I think that so long as prayer is assigned randomly, one can expect the covariates to be distributed randomly across the two groups. In this case, the covariates can be safely ignored.
Click to expand...

Not if they are confounders or endogenous to the matter at hand...

You always need to verify your assumptions...
 
Digithead - Please explain how the control and active groups would differ, based on what you have said.

And you can get a false positive by loading up one of the groups with people who had a less severe form of cancer who thereby benefitted most from their medical treatment rather than prayer but it looks like it was prayer that did it. Hence why you have to risk-adjust in your statistical tests...
Click to expand...

This is obviously spurious, as there is no way to "load up" the groups in such a manner when the assignment is done programatically at random.
 
GK - My protocol proposal is as you see it here; no revisions needed yet. (Except perhaps matters of implementation such as hosting the db with group assignments in a secure neutral location - but those don't really have to do with the protocol per se.)

If you have any suggestions for what should be revised, please tell me.

P.S. "As good as possible"? :P
 
Digithead, wouldn't the problems you describe only emerge if the sample size was too small?

Also, couldn't you protect against such "outlier" problems by, say, discarding some certain percentage (or fixed number) of "top" and "bottom" scorers from each group?

That would also be a way of protecting against the risk that some "prayers" may find a way to contact their "prayees"--or vice versa. (P.S., saizai, do you have a way around this? It's not hard to imagine that someone madly praying for some sick person would be curious to know how that person is doing, and both sick person and praying person know what website they visited to enroll in the study; it wouldn't be too hard to imagine Googling around on the study to see if anybody is blogging about participating etc etc.
 
saizai said:
GK - My protocol proposal is as you see it here; no revisions needed yet. (Except perhaps matters of implementation such as hosting the db with group assignments in a secure neutral location - but those don't really have to do with the protocol per se.)

If you have any suggestions for what should be revised, please tell me.

P.S. "As good as possible"? :P
Click to expand...

I do consider proper double blinding a vital part of the protocol.

I volunteer to serve as a third party. However, this may pose logistical problems since I live in Germany. If you (we) can work out a protocol which allows for electronic data transfer, I'm in the game.

For specific suggestions to the protocol - let me get some sleep first. I'm wiped, I start to see yellow flags where there were none before. "Nightnight, Bingaling."
 
Also BTW, to address what Jeff mischaracterized as well:

I am very explicitly NOT claiming or testing (and have no desire to debate):
* the existence of any deity
* what the mechanism of prayer is, if it works
* any difference between different styles of prayer
* whether any particular Recipient has been prayed for or not, or how much
* the ability to heal any particular Recipient
* any personal ability of mine (I won't be participating except as administrator)
* the effectiveness of any particular treatment other than the prayer gotten through being in the active group

I reserve the right to pick subgroups at my descretion BEFORE the onset of the second/third round, so long as the criteria are made completely explicit. As a hypothetical example, if people who self-identify as Born-Again Christians do amazingly well as Healers in the first round, I may choose to require that Healers in the 2nd and 3rd rounds be BACs as well.

This goes together with the score method choice, and in any case both preceeds the test and could not possibly (by mundane explanation) cause a false positive.
 
Yoink - I think it would be relatively hard to google someone by first name alone.

However, I would be asking both parties to sign a certification stating that they will not, and at the end that they have not, contact any other participant before the end of the round.

GK - I agree wrt double blinding. Would you agree that, if the database containing assignments is both filled and accessed programmatically alone until the end of the experiment (kept encrypted and locked otherwise), and you have source code read access that shows there is no "phone home" feature, that would be sufficient to ensure a complete double blind?

By the start of the second round I would be completely prepared to hand over the website to a trusted third party, to run purely on its own, and require read/write access only to public pages (e.g. to post general news).
 
Yoink - I think it would be relatively hard to google someone by first name alone.
Click to expand...

True, but that wasn't my point. I meant that someone could google "prayer efficacy study" or "prayermatch.org" or "praying from photo alone" etc. etc. and find someone who has left an online record of being involved in the study. You could go onto any chat-type site and say "anyone know anyone involved in a prayer efficacy study"? If you floated this around support-group websites for people with inoperable cancer (the kinds of people who will be drawn to your study, I imagine) you may not have to work as hard as you're thinking to start unblinding the experiment.
 
Yoink - That would take a considerable amount of effort. Given that they are swearing not to do so, swearing that they have NOT done so, and that we will be putting them in contact with each other afterwards if they so desire, I don't see that this is a major flaw. We can eliminate any specific participants who upon investigation (or self-admission) have attempted or succeded in breaching the double blind. I see no reason to apply an a priori culling of the results however, and would not agree to such a requirement.
 
You must log in or register to reply here.

Back
Top Bottom