Remote healing

[sf108]

So the randomization of the prayers from third-parties will valid this as a "controlled" test? hmm...how random are the prayers from outside sources then? Evenly random or random random?

Oh, what about people praying negatively? ie. wishing ill-will to one of the test subjects. Will that negate someone else's prayers? What if a big bloke negatively prayed? Will that negate 2 or 3 other smaller positive prayers?

 

[69dodge]

Sai, I think that people's main complaint can be summarized as follows: the level of significance of your test may be an acceptably low 5%, but what's its power?

The problem with a test of low power is not just that it has a good chance of yielding a false negative. Such a test also has the problem that, even if it should yield a positive result, that positive result would constitute only weak evidence in favor of the effect being tested.

An extreme example: Suppose I ignore the patients' conditions entirely and instead flip a coin ten times. If all ten flips result in heads, I will declare that prayer helps patients recover. Obviously, this is silly. But the level of significance of this test is an impressive-looking p < 0.001, since, on the null hypothesis of the ineffectiveness of prayer, there is less than one chance in a thousand that all ten flips will come up heads. However---and this is the technical reason why it's so silly---the power of the test is ridiculously low: on the alternative hypothesis that prayer is effective, the probability of getting ten heads is still less than one in a thousand. Therefore, even if I do happen to get ten heads, I still cannot conclude anything at all about the effectiveness of prayer.

 

[saizai]

An extreme example: Suppose I ignore the patients' conditions entirely and instead flip a coin ten times. If all ten flips result in heads, I will declare that prayer helps patients recover. Obviously, this is silly. But the level of significance of this test is an impressive-looking p < 0.001, since, on the null hypothesis of the ineffectiveness of prayer, there is less than one chance in a thousand that all ten flips will come up heads. However---and this is the technical reason why it's so silly---the power of the test is ridiculously low: on the alternative hypothesis that prayer is effective, the probability of getting ten heads is still less than one in a thousand. Therefore, even if I do happen to get ten heads, I still cannot conclude anything at all about the effectiveness of prayer.

Um. If you ignore patients and just flip a coin, then how can you say anything at all about patients?

For that matter, what are your control vs. test groups?

I acknowledge that this would produce a false negative if the "power of prayer" is small; that's okay.

 

[69dodge]

Originally posted by saizai
Um. If you ignore patients and just flip a coin, then how can you say anything at all about patients?

I can't. But why not? Isn't the level of significance of my test low, as it should be?

For that matter, what are your control vs. test groups?

The same as in yours. I simply choose to ignore them and to look at my coin instead.

I acknowledge that this would produce a false negative if the "power of prayer" is small; that's okay.

It's not ok. It means that if you get a positive result, there's a good chance that it's a false positive.

 

[saizai]

I can't. But why not? Isn't the level of significance of my test low, as it should be?

A) "high" not "low"
b) No, per below.

The same as in yours. I simply choose to ignore them and to look at my coin instead.

Then your results will be the same - the patients will be statistically identical (minus noise, which decreases inversely with N). So your assertion will be spurious. You can't simply make up whether they get better or not.

It's not ok. It means that if you get a positive result, there's a good chance that it's a false positive.

No, per above.

 

[69dodge]

Originally posted by saizai
A) "high" not "low"
b) No, per below.

I meant "low", as in "p < 0.001". Which it is, in my silly example.

No, per above.

It is true. But I think the extreme silliness of my example is getting in the way. So let's go back to the real example. People in this thread have argued that there will probably be lots of praying going on anyway, for everyone in the study including the control group, and therefore the study probably won't find a difference between the groups even if prayer does have some effect. You don't disagree with this, I think, but you say that it doesn't matter because it only increases the chance of a false negative assuming prayer works but it doesn't increase the chance of a false positive assuming prayer doesn't work. That's all true, except that it does matter anyway. It means that probably you'll get a negative result, which is fine, but it also means that if you happen to get a positive result, you don't really have good reason to believe that the positive result was due to any effectiveness of prayer; it is almost as likely to be simply the result of chance. If a positive result is unlikely assuming prayer doesn't work, and a positive result is also unlikely assuming prayer does work, then why should getting a positive result lead you to believe one possibility over the other?

 

[saizai]

I meant "low", as in "p < 0.001". Which it is, in my silly example.

Low probability. High significance, which is the word you used.

Also, the problem with your example is not that it's silly - I don't care about that - but that it violates experimental rules (and fabricates data). Make as silly an example as you want, but if you make it exactly analagous to my design - that is, only swapping out nouns, not procedure - then it will be equally valid.

It means that probably you'll get a negative result, which is fine, but it also means that if you happen to get a positive result, you don't really have good reason to believe that the positive result was due to any effectiveness of prayer; it is almost as likely to be simply the result of chance. If a positive result is unlikely assuming prayer doesn't work, and a positive result is also unlikely assuming prayer does work, then why should getting a positive result lead you to believe one possibility over the other?

Well, no. It'll be exactly as likely to be due to chance as the significance says. By definition.

A positive result assuming prayer doesn't work - i.e. a false positive - would mean that there actually occured that 5% chance of "results due to chance".

A positive result assuming prayer does work - i.e. a true positive - would mean that it, well, works. With a 95% probability of *not* being due to chance.

Er, that's sorta axiomatic...

 

[69dodge]

Originally posted by saizai
Well, no. It'll be exactly as likely to be due to chance as the significance says. By definition.

By definition, the level of significance is the probability, assuming the null hypothesis, of a positive result. That is not at all the same as the probability, assuming a positive result, of the null hypothesis. In fact, the conventional frequentist approach to statistics considers the latter meaningless. (Which is rather a shame, as it is what we're really interested in. But anyway...)

A positive result assuming prayer doesn't work - i.e. a false positive - would mean that there actually occured that 5% chance of "results due to chance".

A positive result assuming prayer does work - i.e. a true positive - would mean that it, well, works. With a 95% probability of *not* being due to chance.

A positive result assuming prayer doesn't work means that prayer doesn't work. A positive result assuming prayer works means that prayer works. Naturally. The problem is we don't know for sure which of those assumptions is right. All we know for sure is that the result was positive. If prayer has no effect, a positive result was a priori unlikely. If, assuming that prayer does have some effect, the study was anyway a priori fairly unlikely to show a difference between the two groups, due to all the extra-study prayer that probably went on, then the positive result doesn't much help us to decide whether prayer works or not: either way, what ended up happening was a priori unlikely.

 

[saizai]

By definition, the level of significance is the probability, assuming the null hypothesis, of a positive result. That is not at all the same as the probability, assuming a positive result, of the null hypothesis. In fact, the conventional frequentist approach to statistics considers the latter meaningless. (Which is rather a shame, as it is what we're really interested in. But anyway...)A positive result assuming prayer doesn't work means that prayer doesn't work. A positive result assuming prayer works means that prayer works. Naturally. The problem is we don't know for sure which of those assumptions is right. All we know for sure is that the result was positive. If prayer has no effect, a positive result was a priori unlikely. If, assuming that prayer does have some effect, the study was anyway a priori fairly unlikely to show a difference between the two groups, due to all the extra-study prayer that probably went on, then the positive result doesn't much help us to decide whether prayer works or not: either way, what ended up happening was a priori unlikely.

How about this: make the same argument, but using (say) an equivalently designed test to determine whether administering psuedophedrine reduces the symptoms of the common cold (e.g. runny nose etc).

 

[Gayle]

Intellectual dishonesty

How about this: make the same argument, but using (say) an equivalently designed test to determine whether administering psuedophedrine reduces the symptoms of the common cold (e.g. runny nose etc).

Sai, you're not being logical.

In your own words, you're just going to ask for certain people to be prayed for or not prayed for. It's "purely superfluous" -- your words -- to check to see if the prayers actually take place.

Your analogy with pseudophedrine would go something like this: an equivalently designed test to determine whether asking someone to think about administering psuedophedrine reduces the symptoms of the common cold (e.g. runny nose etc) in another person.

You're also being extremely naive about the difficulties and costs of running such an experimental design on several hundred patients. These kinds of large scale distant healing medical experiments cost hundreds of thousands of dollars.

funding for Ms. Targ to conduct a three-year study of distant healing on 150 HIV patients. The funding for the first year alone is $243,228...

http://www.csicop.org/si/2001-03/fringe-watcher.html

The cost of copying medical records alone runs into thousands of dollars. Hospitals, labs and doctors do not provide such services for free.

If you knew anything about these kinds of experiments, you would know that from doing a basic review of the literature. You haven't done your homework.

 

[saizai]

Re: Intellectual dishonesty

Sai, you're not being logical.

In your own words, you're just going to ask for certain people to be prayed for or not prayed for. It's "purely superfluous" -- your words -- to check to see if the prayers actually take place.

Your analogy with pseudophedrine would go something like this: an equivalently designed test to determine whether asking someone to think about administering psuedophedrine reduces the symptoms of the common cold (e.g. runny nose etc) in another person.

Well, not really. Because of the double blind, it is exactly analagous to administering any other treatment which the patient cannot tell from the placebo - like psuedophedrine vs. sugar pill.

You can argue that it will have no effect, of course - and I expect you to do so - but methodologically it's equivalent. And in any case, your only avenue for arguing this is "no known avenue of effect", trumped by an actual testing (which is the whole point of this, ya?).

(Also re. "purely superfluous" - it is, from the point of view of a skeptic [since either version is double-blind with no mundane avenue of effect]. Obviously not, from a believer's POV.)

You're also being extremely naive about the difficulties and costs of running such an experimental design on several hundred patients. These kinds of large scale distant healing medical experiments cost hundreds of thousands of dollars.

http://www.csicop.org/si/2001-03/fringe-watcher.html

The cost of copying medical records alone runs into thousands of dollars. Hospitals, labs and doctors do not provide such services for free.

If you knew anything about these kinds of experiments, you would know that from doing a basic review of the literature. You haven't done your homework. [/B]

Not naive so much as optimistic. I don't consider it a barrier to trying, or to coming up with a good design first.

In any case, that's my problem not yours.

 

[69dodge]

Originally posted by saizai
How about this: make the same argument, but using (say) an equivalently designed test to determine whether administering psuedophedrine reduces the symptoms of the common cold (e.g. runny nose etc).

If participants, including those in the control group, were likely to get some of the same drug through outside channels, the same argument would apply: not only would a positive result be less likely, but also, even if it were to happen it wouldn't provide much evidence in favor of the drug's effectiveness.

The point of the control group, after all, is that its members don't get the treatment being tested. If you have two groups, and they both get treated, then first, the study will probably not find a large difference between the groups, and second, in the unlikely event that it does find a much larger improvement in one group than in the other, there's no reason to attribute that improvement to the treatment, because the other group was treated too. The appropriate conclusion would be, sure it was unlikely that there would be a big difference, but sometimes unlikely things do occur purely by chance, and this happens to be one of those times.

 

[saizai]

If participants, including those in the control group, were likely to get some of the same drug through outside channels, the same argument would apply: not only would a positive result be less likely, but also, even if it were to happen it wouldn't provide much evidence in favor of the drug's effectiveness.

I would disagree. It's easier to demonstrate this for something not a drug - since drug tests are usually for a substance unavailable other than through the study.

Take, say, talk psychotherapy. There's no way to control for that - people talk to their friends etc. all the time. But you can say, what is the effect of the therapy that *we* provide them?

The point of the control group is not quite that they don't get the treatment being tested - it's that they are treated differently from the test group. It's relatively rare that you can have a fully "clean slate" test; there will always be outside influences, often of the very thing you're testing.

Another example: dentist-office fluoride treatments.

Everybody in America is exposed to fluoride in drinking water. Everyone. And in varying amounts, depending on their consumption.

But that doesn't mean you can't still test to see whether a fluoride paste treatment provides benefits, over a control group who get neutral paste and are exposed to the same environment.

You can always argue that "a little is enough" - i.e. that there's no benefit to additional treatment - but this would be proved easily enough by the same study, in conjunction with some test to prove that the baseline does in fact include some treatment.

But you're making a (very understandable) confusion - that to tell the difference between (baseline) and (baseline+treatment), you require (baseline) not to include (treatment). You don't.

 

[69dodge]

Originally posted by saizai
You can always argue that "a little is enough" - i.e. that there's no benefit to additional treatment - but this would be proved easily enough by the same study, [...]

It's not as simple as that. If someone, based on all his previous experience with flouride, had, before this particular study was done, a very strong belief that "a little is enough", he would not be convinced that "a little isn't enough" even if the study returned a positive result. He would argue that the most likely explanation of the positive result is that it occurred purely by chance. Sure, the probability of that happening was only 1 in 20, but in his opinion the probability that "a little isn't enough" was even less than 1 in 20, perhaps much, much less.

Such a position is entirely reasonable. One could argue about whether he was justified in his prior estimate of the probability that "a little is enough". But that has nothing to do with this study. Given his prior estimate, his interpretation of this study's positive result is faultless.

What we should believe after a study depends in part on the results of the study. It also depends in part on what we believed before the study. It's incorrect simply to ignore that aspect of things. So let's consider what we believe, before any studies, about the effectiveness of prayer. Prayer by those who will take part in the study has some probability of making a noticeable difference between the groups. I think the probability is rather small. I guess you think it's somewhat larger. But in any case, we can both agree that if other people will be praying for those in the control group too, the probability of prayer making a noticeable difference between the groups is smaller than if no such outside prayer went on. Therefore, the same holds true after a positive study: if other people probably prayed for those in the control group, our estimate of the probability that prayer works ought to be less than if no other praying went on.

You can't just change the subject of the study, from "is prayer effective?" to "is prayer by specific people effective, even in the presence of lots of other prayer?", without taking into account the fact that the latter hypothesis has a smaller prior probability than the former.

To put it bluntly, if one studies a sufficiently silly hypothesis, any positive result of the study is more likely to be due to chance than to the silly hypothesis actually being true.

I'm not really prepared right now to argue about exactly how silly prayer is. All I'm saying is that if someone thinks it is very silly---more precisely, if someone thinks that its probability of working is much less than 5%---he will not be convinced by a study whose level of significance is 5%. And he is right not to be convinced.

 

[saizai]

It's not as simple as that. If someone, based on all his previous experience with flouride, had, before this particular study was done, a very strong belief that "a little is enough", he would not be convinced that "a little isn't enough" even if the study returned a positive result. He would argue that the most likely explanation of the positive result is that it occurred purely by chance. Sure, the probability of that happening was only 1 in 20, but in his opinion the probability that "a little isn't enough" was even less than 1 in 20, perhaps much, much less.

Such a position is entirely reasonable. One could argue about whether he was justified in his prior estimate of the probability that "a little is enough". But that has nothing to do with this study. Given his prior estimate, his interpretation of this study's positive result is faultless.

I would strongly disagree.

To put it bluntly... what you believe is utterly irrelevant to the interpretation of the results. If the significance says that the probability of error (i.e. results due to chance) is 5%, then it's 5%. Your critical perspective is not a data point; your handwave of some "prior probability" is purely that - a handwave. Relevant only in terms of your psychology, not science.

You could argue that 5% is not enough, and call for a repeat or a study with higher significance threshold, but this would simply drive that 5% down (assuming the successive ones are correct). You must, if you are to retain self-consistency as a skeptic, be willing to assign a priori some level of significance at which you will believe the results, no matter what your previous belief on the subject.

If you don't, then you're no longer logical - your position becomes "I don't belive it, so nyah", no matter how convincing the evidence - e.g. if the significance were 99.999%.

As for the "a little is enough" - a positive would not prove either way (for one could say that some people in both groups didn't get any, and thus that only those improved - but that would be shown in the results too); a negative would prove that either "a little is enough" or the null, that prayer doesn't work, or some other confound. All, of course, to the same degree of significance.

To put it bluntly, if one studies a sufficiently silly hypothesis, any positive result of the study is more likely to be due to chance than to the silly hypothesis actually being true.

This way, as explained above, lies dogmatic disbelief.

(I would like to reiterate, FWIW, that I make no claims about how, why, mechanism, etc., for this to work. I would suggest that you do likewise; you can state your a priori belief as a skeptic that it doesn't since it has no known avenue, but you have no foundation on which to argue - again, as a skeptic - anything whatsoever concerning how it would work or what its limits might be. That would require you to assume the major point to be conceded - i.e. that prayer works at all.)

Actually, I should re-reiterate that last point:

You don't get to make the argument "if prayer works in way X, then you will get a false positive". Because, if it does work in way X, then it works... which is a true positive. And if it doesn't, then the argument is vacuous. You can certainly argue for a false negative, and I think I've already agreed that that might be the case; a necessary risk, given the vast number of (mutually exclusive) theological frameworks available in this world.

Nor do you get to make the argument that prayer *only* works in way X - none of us do, since we have no grounds for that.


All that said, I am highly curious to hear your logical basis behind the argument you just gave. You seem quite convinced in it - though I see it as fallacious at core - and I would rather like to understand why.

 

[Elaedith]

I would strongly disagree.

To put it bluntly... what you believe is utterly irrelevant to the interpretation of the results. If the significance says that the probability of error (i.e. results due to chance) is 5%, then it's 5%. Your critical perspective is not a data point; your handwave of some "prior probability" is purely that - a handwave. Relevant only in terms of your psychology, not science.

..............

All that said, I am highly curious to hear your logical basis behind the argument you just gave. You seem quite convinced in it - though I see it as fallacious at core - and I would rather like to understand why.

It sounds to me as though the previous poster is using logic similar to that of Bayes thereom, which is hardly 'hand-waving'. I gave some links in your other thread to criticisms of conventional or Fisherian null-hypothesis testing, (NHT) which seems to be the model you are proposing to use. One criticism of NHT is that it fails to take account of the prior probabilities in reaching conclusions based on statistical tests. The outcome of NHT does not actually tell you the probability that the null hypthesis (that chance alone operates) is true. It tells you the probability of getting the effect you have found if the null hypothesis is true. In other words, the logic:

If the null hypothesis is true, an effect is unlikely
An effect has occurred
Therefore, the null hypothesis is unlikely

is invalid reasoning.

It is not true that setting a higher alpha level than .05 provides stronger support for the conceptual hypothesis. Using a higher alpha value reduces the risk of rejecting the null hypothesis if it is true, but that still doesn't tell you the probability of the null hypothesis being true.

Now some critics of null hypothesis testing do believe that prior probabilities should be employed along Bayesian lines in statistical testing. An example if given in the following link which is the first one I found, although there is plenty more about this online:

http://ourworld.compuserve.com/homepages/rajm/twooesef.htm

Although the majority of researchers in social, behavioural and health sciences do currently use conventional hypothesis testing rather than the Bayesian approach, there is a sense in which prior probabilities are considered, because conclusions in these areas are usually reached on the basis of the preponderance of evidence from converging independent sources. In other words, if somebody obtains a result in a single test that appears to go against the current weight of evidence, the entire current body of knowledge is not going to be abandoned because of this one result. Conceptions of the world will only change when the bulk of the evidence supports such as change. The Bayesian approach simply states that we examine current knowledge to determine the plausibility of a hypothesis and assign it a prior probability explicity before commencing.

Going back to random assignment: while it should equalise the effect of all variables other than the manipulated one, it cannot guarantee to do so. Most research does not rely solely on randomisation for control, but actually tries to eliminate potential confounds as much as possible, by conducting research under laboratory conditions which prevent these confounds from happening. When this is not possible, attempts are usually made to measure the the incidence of potential confounds to see if they actually were equal for both groups during the study. Failing all that, an experiment with a high likelihood of confounds and only randomisation for control would provide only weak support for a prediction even if the null hypothesis is rejected.

But in cases such as medical research, before a treatment gets anywhere near being tested on real people in a naturalised context, it will have already been extensively tested through other means - eg. tissue samples, animal studies, lab studies, simulation. In other words, the treatment likely has a high prior probability of being effective, and this probability will affect interpretion of the results, regardless of the statistical paradigm used.

So - most people will agree that the intervention you propose has a low prior probability of being effective based on our current understanding of the world and the current evidence. You have no experimental control other than to hope that all other variables will be the same for both groups due to randomisation. You haven't done any previous tests. Therefore, it is unlikely that your results would convince anyone. The same would be true if you were testing any other treatment under similar conditions. However, if the weight of the evidence after many replications started to favour your hypothesis, then the situation would change, although evidence from only one source would not be sufficient. The more skeptical someone is, the more evidence they are likely to require, but this is true of all research.

 

[69dodge]

Originally posted by saizai
If the significance says that the probability of error (i.e. results due to chance) is 5%, then it's 5%.

If the level of significance is 5%, this means that, assuming prayer has no effect, the probability is 5% that a future study will show a positive result anyway purely by chance.

But that probability is not directly relevant. We don't know whether prayer has an effect or not; that's what we're trying to determine. Rather, what we know is that the study showed a positive result (supposing that it did). Now, the question we're faced with is, given the study's positive result, what's the probability that prayer is not effective and that the positive result was due just to chance? This is not the same question, although it sounds the same ("what's the probability that a positive result is due to chance?"), because the assumptions are different. Before, we assumed that prayer was ineffective and we didn't yet know the result of the study; now, we know the result of the study but we don't know whether prayer is effective.

It is a different question, and in general it has a different answer. The answer does depend on the level of significance of the test and its power, but it also depends on the prior probability of the effectiveness of prayer.

You could argue that 5% is not enough, and call for a repeat or a study with higher significance threshold, but this would simply drive that 5% down (assuming the successive ones are correct). You must, if you are to retain self-consistency as a skeptic, be willing to assign a priori some level of significance at which you will believe the results, no matter what your previous belief on the subject.

I should be able to decide on a level of significance that would convince me, but the level I pick will depend on my previous belief. The surer I previously am that prayer doesn't work, the stronger evidence of its effectiveness I will require before I change my mind. That makes sense, no?

And if I'm more sure that prayer by designated healers won't result in a difference between the test group and the control group when others might be praying for the control group than when it's certain that no others are praying for them, which I am (aren't you?), then I'm justified in requiring stronger evidence for the former hypothesis than the latter.

69dodge: To put it bluntly, if one studies a sufficiently silly hypothesis, any positive result of the study is more likely to be due to chance than to the silly hypothesis actually being true.

saizai: This way, as explained above, lies dogmatic disbelief.

Yes, I wasn't very precise. I don't mean that one should ever simply ignore a study. But, depending on one's prior belief and the power and level of significance of the study, a positive result might only have the effect of changing one's opinion about the hypothesis tested from 'almost certainly false' to 'very probably false', rather than from 'probably false' to 'probably true'.

(I would like to reiterate, FWIW, that I make no claims about how, why, mechanism, etc., for this to work. I would suggest that you do likewise; you can state your a priori belief as a skeptic that it doesn't since it has no known avenue, but you have no foundation on which to argue - again, as a skeptic - anything whatsoever concerning how it would work or what its limits might be. That would require you to assume the major point to be conceded - i.e. that prayer works at all.)

We shouldn't assume that it works. But we need to make some assumptions about how it would work if it did work, so that we can devise an appropriate study to test it. If we don't decide what it is we're looking for, we won't know whether we've found it or not.

You don't get to make the argument "if prayer works in way X, then you will get a false positive". Because, if it does work in way X, then it works... which is a true positive. And if it doesn't, then the argument is vacuous. You can certainly argue for a false negative, and I think I've already agreed that that might be the case; a necessary risk, given the vast number of (mutually exclusive) theological frameworks available in this world.

I agree that if prayer works, a false positive is impossible by definition. But that's not the argument. The argument is that if, on the assumption that prayer is (in some sense) effective, a false negative is nevertheless very likely, then an actual positive result provides only weak evidence in favor of prayer's effectiveness.

Here's one way to think about it: if, on the assumption that prayer works, the study is anyway likely to result in a false negative, then the study isn't really studying prayer, to any great extent. Other unknown factors have much more of an effect than prayer on the study's outcome. Therefore, a positive result from the study doesn't tell us much about prayer, either, because mostly it wasn't a study of prayer to begin with.

The extreme case is a study in which the probability of a negative result is the same whether prayer works or not, as in my previous example where the outcome of a coin flip is taken to be the result of the study. Obviously, in such a case, one can conclude nothing about prayer from the study, because the study was designed in such a way that prayer could have no effect on its outcome. Similarly, if a study is designed so that prayer could possibly affect its outcome, but is unlikely to---i.e., there is a high probability of a false negative---then one can conclude something about prayer from the study, but not much.

All that said, I am highly curious to hear your logical basis behind the argument you just gave. You seem quite convinced in it - though I see it as fallacious at core - and I would rather like to understand why.

It's standard Bayesian reasoning.

http://www.trinity.edu/cbrown/science/bayes.pdf

http://en.wikipedia.org/wiki/Bayesian_inference

 

[saizai]

BBL

Classes have started (yay college), so this (along with all my other low-priority stuff) is on pause until I have more time on my hands.

Be back eventually.

*goes back to reading neuroscience articles*

 

[Octothorpe]

Sai said:

It's easier to demonstrate this for something not a drug - since drug tests are usually for a substance unavailable other than through the study.

I think this is exactly the problem. You are doing the equivilant of testing the effectiveness of a drug, but the supply of the drug is not controlled.

I know what you mean about the effects of third party prayer not coming into play because of randomization between the two groups. I really do get that. I just worry that the third party prayers introduce so much noise into your data that it is going to be impossible to pick out any effect your experiment may have.

Lets take the drug situation. We have a few hundred sick people who are divided into two groups. Once a week, half the people are given a pill with a drug in it, and half are given a pill with sugar in it. (Bear with me, I'm not as stupid as I sound, nor do I assume you are stupid either. I just like to start at the beginning.)

But, this drug is not a controlled substance and anyone can get any amount they want at any time. One sick person has a husband who is giving her one pill a minute! One guy gets 5000 pills every Sunday morning! One guy gets nothing. One person has every man, woman and child in the state of Indiana sending him random numbers of pills at random times. One person gets pills from his brother 20 times a day for 2 weeks, then nothing after that. One person gets 4 pills a day, one of which is the drug we are testing, and three which come from China and are similar but not quite the same. Every person in the experiment has a different situation with respect to this drug. (Ok, I know you get the point. Sorry to belabor.)

It will be very complicated for the drug makers to filter out all of this noise and see if their one pill a week had any effect. I'm sure its not impossible, given a really large study population. But how big does that population have to be? It seems to me it would have to be very large indeed to see such a tiny effect. Maybe somebody out there with more statistical math than I have can figure this out. Maybe somebody from the SETI project. No joke intended, they are used to processing vast amounts of data for tiny aberations. I think this study may be on that scale.