Need Help with Randomizing for Experiment

[JJM]

{snip} In class experiments with 20 subjects, I use a deck of 20 cards, 10 red and 10 black suits, shuffle them and assign conditions based on which color they draw.

That sounds good. I would have suggested labeling cards 'P' for "prayer" and 'P' for "plain."

{snip} I'll be running the experiment because I don't come in contact with this guy; he works with Mr. Amapola and is more his friend than mine. I only see the guy very occasionally, and not up here where we live. So for this reason, I won't tell Mr. Amapola to which group each pot belongs. {snip}

You definitely have to get the believer to come up and determine the outcome, all by himself. Whatever criterion, or criteria, for efficacy must be specified before he makes his choices. The more criteria he chooses, the more likely it is he will find a difference, and modified, more-rigorous statistics have to be applied.

For the statistically-challenged (e.g., me) it is best to choose one criterion. On the other hand, that allows the true believer to rationalize that he simply chose the wrong measure. And that is what believers do in the face of negative results- they rationalize (see Randi's book "Flim-Flam").

{snip} Do you really need to go through the process of blinding and randomization then? If he is not involved in viewing the experiment, you don't have to prove anything to him. And I agree that two groups of 10 are fine if it's just you looking at the plants (that extra layer of uncertainty is not necessary).

Linda

I agree; but the application of randomization is simple, and it removes one avenue of post-hoc rationalization.

 

[HawaiiBigSis]

<snip>The more I think about this the more convinced I become that the guy will drop out and refuse to say which plants belong in which group before the end. Maybe I need more faith in his faith.

In the end, you'll get a bunch of beans, which is a nice enough reward for his faith, or lack thereof, yeah?

Have you decided how long to run the test? All the way to fruit, or will you stop sometime before that? Maybe the number of beans per plant would be a determination factor that eliminates the need to decide whether any particular plant is healthier than another.

 

[Jeff Corey]

In the end, you'll get a bunch of beans, which is a nice enough reward for his faith, or lack thereof, yeah?
"It ain't worth a hill of beans" might apply here.

 

[Hokulele]

The way Dr. Corey describes is the way I had planned to do it - let the guy walk around and look at all the plants, and say one by one which plant belongs in which group. When he was finished, then I would produce the map that showed which plants were given the prayed for water and which the normal water. With only me knowing which were which, I would probably have my husband go around with him and record his answers.

69dodge, I'll check out that link! That's a nice high-tech way to get the numbers.

The more I think about this the more convinced I become that the guy will drop out and refuse to say which plants belong in which group before the end. Maybe I need more faith in his faith.

I vote that you mess with his head. Have only 1 pot use the prayed for water and see if he can pick it out.

 

[Amapola]

In the end, you'll get a bunch of beans, which is a nice enough reward for his faith, or lack thereof, yeah?
"It ain't worth a hill of beans" might apply here.

Except it will be... and a hill of beans might be all I end up with!

I vote that you mess with his head. Have only 1 pot use the prayed for water and see if he can pick it out.

You are evil. I like it.

However I will stick to what I have agreed to do which is water half of them with holy water. I'm such a boring person...

 

[69dodge]

I am assuming that he will make all his guesses in one sitting and not be given feedback until all guesses are recorded. This would make each guess independent of the previous ones, especially if he were not told how many were in each group.
So a Binomial or Sign test should be appropriate.

Right, he won't actually be given feedback until he's done making all his guesses. But the various guesses still aren't independent, because, if he were given feedback on earlier guesses, he would be able to do better on later ones. Which is why he shouldn't be given feedback.

That's for an experiment where everyone knows that there are sure to be ten plants in each group. I think this is what Amapola intends to do.

But a binomial test is appropriate if each plant is independently assigned to a group, as by a coin flip, so that the two groups might not end up being the same size. (fls suggested doing the experiment this way.) And, in this case, it's perfectly ok to give him feedback after guessing the status of each plant, because doing so won't help him make better guesses about subsequent plants.

 

[bpesta22]

Right, he won't actually be given feedback until he's done making all his guesses. But the various guesses still aren't independent, because, if he were given feedback on earlier guesses, he would be able to do better on later ones. Which is why he shouldn't be given feedback.

That's for an experiment where everyone knows that there are sure to be ten plants in each group. I think this is what Amapola intends to do.

But a binomial test is appropriate if each plant is independently assigned to a group, as by a coin flip, so that the two groups might not end up being the same size. (fls suggested doing the experiment this way.) And, in this case, it's perfectly ok to give him feedback after guessing the status of each plant, because doing so won't help him make better guesses about subsequent plants.

I think giving him feedback after each guess would bias the test in the way you argued above.

With no feedback, we likely have to agree to disagree on our probability arguments (though I submit the independence assumption applies only to the P correct of each trial, which would nt be violated unless they guy got feedback after each -- for example, the p of getting trial 20 correct would be 1.0).

 

[69dodge]

I submit the independence assumption applies only to the P correct of each trial, which would nt be violated unless they guy got feedback after each -- for example, the p of getting trial 20 correct would be 1.0.

Independence of events doesn't mean that they have the same probability. Two events with the same probability might be independent or not. Two events with different probabilities might be independent or not.

By definition, two events are independent if the probability that both occur together is equal to the product of their individual probabilities.

Instead of twenty plants that are split ten and ten, suppose there are only two, one prayed for and one not. Either the first was and the second wasn't, or vice versa. The guy knows that these are the only two possibilities, and will guess one of them. The probability is 1/2 that he guesses correctly about the first plant. The probability is also 1/2 that he guesses correctly about the second plant. If the two events (correct guess about plant 1, and correct guess about plant 2) were independent, the probability that both his guesses are simultaneously correct would be 1/4. The binomial distribution, which assumes independence, says it's 1/4. But it isn't 1/4. It's 1/2. Either he gets both plants right, or both plants wrong. There's no chance that he'll get one right and one wrong, whereas, according to the binomial distribution, such an event should have probability 1/2.

 

[bpesta22]

Independence of events doesn't mean that they have the same probability. Two events with the same probability might be independent or not. Two events with different probabilities might be independent or not.

By definition, two events are independent if the probability that both occur together is equal to the product of their individual probabilities.

Instead of twenty plants that are split ten and ten, suppose there are only two, one prayed for and one not. Either the first was and the second wasn't, or vice versa. The guy knows that these are the only two possibilities, and will guess one of them. The probability is 1/2 that he guesses correctly about the first plant. The probability is also 1/2 that he guesses correctly about the second plant. If the two events (correct guess about plant 1, and correct guess about plant 2) were independent, the probability that both his guesses are simultaneously correct would be 1/4. The binomial distribution, which assumes independence, says it's 1/4. But it isn't 1/4. It's 1/2. Either he gets both plants right, or both plants wrong. There's no chance that he'll get one right and one wrong, whereas, according to the binomial distribution, such an event should have probability 1/2.

Dodge, thanks, but I still disagree (which is why I said perhaps we need to agree to disagree). With just two scenarios-- and no feedback-- the probability is 1/4 that he'd get both right (or both wrong. and, it's 1/2 that he'd get at least one right.

With feedback, then we leave the realm of probability for trial 2 and use deductive certainty to guarantee that trial is correct. With feedback and only 2 trials, assuming the selector is rational, it's a 50/50 chance, as you said.

With feedback and 20 trials, you could use deduction to get at least the last trial correct for sure, and maybe a few more (the maximum you could get right deductively would be 10, in the odd chance all of one type were ordered first).

That's why I said don't give feedback, and I think we agree on the points above.

Without feedback, the probability on each trial is independent. That you *guessed* prayed for 10 times already by trial 20 has no affect on the *probability* that trial 20 is a prayed for bean.

I'm saying, without feedback, no possible guessing strategy would make it so the probability of getting it right on this trial depends on the probability of getting it right on any other trial.

This I think is where we disagree-- the independence isn't concerned with how people select on each trial, but only with the probability of success on each trial (which I claim cannot vary by guessing strategy, unless the selector gets feedback).

B

 

[Jekyll]

Dodge, thanks, but I still disagree (which is why I said perhaps we need to agree to disagree). With just two scenarios-- and no feedback-- the probability is 1/4 that he'd get both right (or both wrong. and, it's 1/2 that he'd get at least one right.

Is your definition of feedback that the guesser knows how many of the total plants have been given holy water or something else?

Assuming the guesser already knows that only one of the two plants has been watered, there are only two guesses he can legitimately make: That the first plant was watered or the second plant was watered. So he has a 50% chance of being 100% right, without any additional information being offered.

 

[bpesta22]

Is your definition of feedback that the guesser knows how many of the total plants have been given holy water or something else?

Assuming the guesser already knows that only one of the two plants has been watered, there are only two guesses he can legitimately make: That the first plant was watered or the second plant was watered. So he has a 50% chance of being 100% right, without any additional information being offered.

I agree on the 2 trial scenario where the selector is told whether his first pick is right before he makes the second pick.

By feedback in the 20 trial scenario, I meant that after each pick, he is told whether he's right or wrong. This would violate the independence assumption for at least the last trial.


Without such feedback, my assertion is that this is squarely the binomial distribution with no violation of the independence assumption.

Knowing that 10 are prayed for and 10 are not does not violate the independence assumption.

 

[Jekyll]

I agree on the 2 trial scenario where the selector is told whether his first pick is right before he makes the second pick.

By feedback in the 20 trial scenario, I meant that after each pick, he is told whether he's right or wrong. This would violate the independence assumption for at least the last trial.

Ok, so if we stick with the 2 plant scenario, assume the selector knows that there is only 1 holy plant and nothing else, he is given no further feedback, what would his guesses look like?

I can only see him making 2 possible guesses:

"Plant one is holy and plant two is not."
or
"Plant two is holy and plant one is not."

But if you think the binomial distribution applies, you must think he can some how make a choice like:
"Both plant one and two are holy."
or,
"Neither plant one nor two are holy."

as these are the only way to get the 50% right answers, predicted by the binomial distribution.

 

[bpesta22]

Ok, so if we stick with the 2 plant scenario, assume the selector knows that there is only 1 holy plant and nothing else, he is given no further feedback, what would his guesses look like?

I can only see him making 2 possible guesses:

"Plant one is holy and plant two is not."
or
"Plant two is holy and plant one is not."

But if you think the binomial distribution applies, you must think he can some how make a choice like:
"Both plant one and two are holy."
or,
"Neither plant one nor two are holy."

as these are the only way to get the 50% right answers, predicted by the binomial distribution.

Perhaps we're agreeing here but failing to communicate.

I agree that a 2 trial experiment is problematic when one gets feedback (or knows that one is prayed for and one is not). I don't think one can use the same deductions in a 20 trial experiment, with no feedback provided, even if the selector knows that half are prayed for and half are not.

I also think the 2 trial example is logically reduced to a one trial scenario, so I think it is binomial, with p = .5 of getting it right.

 

[Jekyll]

Perhaps we're agreeing here but failing to communicate.

I agree that a 2 trial experiment is problematic when one gets feedback (or knows that one is prayed for and one is not). I don't think one can use the same deductions in a 20 trial experiment, with no feedback provided, even if the selector knows that half are prayed for and half are not.

I also think the 2 trial example is logically reduced to a one trial scenario, so I think it is binomial, with p = .5 of getting it right.

I agree, except with the bolded bit.

It is clear to me that the independence assumption is violated in the 20 case scenario, just as it is in the two case scenario.
If we fix the status of the first 19 plants there is only one possible state that the 20th plant can be in. Similarly if we fix the first 18 plants there are at most two different states the remaining two plants can be in.

If the independence assumptions held there would be 2 possible states the last plant could be in after fixing the first 19 of them, and 4 possible states for the last 2 plants after fixing 18.

 

[Amapola]

OK - bpesta, 69Dodge and Jekyll - am I thinking about this wrong? To me, there will be 10 prayed-for pots. So if he can guess all 10 prayed for ones, he gets 10 out of 10. But if he only guesses 5 of the prayed for ones, he gets 5 out of 10. I'm leaving the other 10, the normal ones or the control group, out of his correct/not correct guesses. That's the way I had planned to record the results. (So I would see a trial with only two pots as he either gets it right or wrong - he either chooses the correct prayed for pot, or chooses the wrong pot.)

I'm striving to set this up so that he gets no feedback as he is guessing, not even unconscious signals from Mr. Amapola.

But am I wrong in only considering the 10 that he will have prayed for?

If this is not the right way to go about it I will of course change it.

 

[bpesta22]

I agree, except with the bolded bit.

It is clear to me that the independence assumption is violated in the 20 case scenario, just as it is in the two case scenario.
If we fix the status of the first 19 plants there is only one possible state that the 20th plant can be in. Similarly if we fix the first 18 plants there are at most two different states the remaining two plants can be in.

If the independence assumptions held there would be 2 possible states the last plant could be in after fixing the first 19 of them, and 4 possible states for the last 2 plants after fixing 18.

Why not fix the states of the last 19 plants and complain about the status of the first one? Or , why not fix the status of 1-9 and 11-20 and claim 10 is violated?

It's clear that there's n-1 degrees of freedom here, but show me how just knowing that changes anything?

Of all possible states for the last plant, half of them include it being prayed for, and the other half include it not being prayed for. p=w/t = .50 for each of 20 trials in the population (for every scenario where the last plant must be prayed for there are equally probably scenarios where the last plant must be a control).

Show me how just knowing that it's broken down 10 and 10 alone can give the selector an advantage where the null is not .50 for any trial.

It seems like the pop quiz fallacy? The pop quiz cant be on the last day of the class, because it wouldn't then be a surprise, etc.

I could be wrong and would concede if someone could explain how just knowing that it's 10 and 10 gives you any advantage whatsover or clues you in on what to guess for pot 20?

 

[bpesta22]

Amp

I think wrong is too strong a word as it seems like either approach would yield the same conclusions except in a very improbable borderline case.

I think it's an academic debate at this point; still submitting that it's a 20 trial binomial test-- not a 10 trial one on just the prayed for beans.

Consider, we agreed that 8/10 would pass in your version and 14/20 in my version.

In one, your subject needs 80% accuracy to win. In the other, just 70% accuracy does it!

As JC suggested, if we make it 15 of 20 to pass, then that's just 75% accuracy, but that's considerably less probable then the 80% accuracy rate needed to pass the 10 trial test. Edited: so it would be easier to pass the 80% accuracy test than the 75% accuracy test!

So, there's psychological advantage to going with n=20. "C'mon you just need 70% (or 75%) accuracy to win! That's like getting a C in a class. Is prayer that weak that it can't even produce a C?"

 

[69dodge]

OK - bpesta, 69Dodge and Jekyll - am I thinking about this wrong? To me, there will be 10 prayed-for pots. So if he can guess all 10 prayed for ones, he gets 10 out of 10. But if he only guesses 5 of the prayed for ones, he gets 5 out of 10. I'm leaving the other 10, the normal ones or the control group, out of his correct/not correct guesses. That's the way I had planned to record the results. (So I would see a trial with only two pots as he either gets it right or wrong - he either chooses the correct prayed for pot, or chooses the wrong pot.)

I'm striving to set this up so that he gets no feedback as he is guessing, not even unconscious signals from Mr. Amapola.

But am I wrong in only considering the 10 that he will have prayed for?

If this is not the right way to go about it I will of course change it.

That's fine.

The number of correct guesses about normal pots will always be the same as the number of correct guesses about prayed-for pots. So there's no need to record both of them explicitly.

(For example, suppose he correctly guesses, about eight of the prayed-for pots, that they were prayed for. Then, he's got two more "prayed-for" guesses left. They will necessarily be made about two of the normal pots. And so, the remaining eight normal pots will be guessed correctly as being normal.)

 

[Jekyll]

..snip..

It's clear that there's n-1 degrees of freedom here, but show me how just knowing that changes anything?
..snip..

Actually, it's more restrictive than that.

Show me how just knowing that it's broken down 10 and 10 alone can give the selector an advantage where the null is not .50 for any trial.

Ok, so under the binomial distribution there are 2^20 = 1048576 possible guesses he can make about which pots are being prayed for. Only one of these guesses will be entirely right.

If we restrict ourselves to just the cases where there are 10 prayed for plants there are ²⁰C₁₀ =184756 possible guesses.
Still exactly one of these guesses will be right.

So if the guesser knows that there are exactly 10 pots being prayed for he is almost 6 times more likely to be completely right than the binomial distribution leads us to believe.

 

[BobK]

Given the small plant population, I wouldn't use normal water at all.

Use the assumption that prayed water is better than normal, and cursed water is worse than or equal to normal.

If there is a difference in growth, it should be most noticeable just using prayed and cursed water.

If the results happen to show cursed as being best, it might put your friend in a state of cognitive dissonance.