Need Help with Randomizing for Experiment

[bpesta22]

Yeah, I saw that too late to edit it. It's Saturday AM, after all.

Morning p's are more significant than p's any other time of the day.

 

[Amapola]

OK, so it sounds like I should drop the cursing and just go with prayed for/not prayed for water. I can change that now easily because we can't begin until the plants are up and it will take 10 days to 2 weeks for them to germinate.

I won't be able to change them around once the plants are growing. These are pole beans; they will quickly grow about 6 - 10' high on a trellis. Yes, they are all right next to each other so should receive the same approximate growing conditions. There is no real concern about the plants being messed with (except that *Claus* has already started praying! ) because I live way up in the mountains in such an isolated area. The guy who believes in the prayer lives about 70 miles away. I would know if he came up here, as he would have to ask for directions!

Actually we are performing the experiment at his insistence, because we (Mr. Amapola and I) do *NOT* believe in prayer and he does. He is convinced that we will see the light as this experiment progresses. He thought it would be OK to use only two plants but I pointed out this was not enough. Unfortunately, I don't have the resources to use 393 plants in each group!

As to how we are supposed to keep the water: he has asked us to get a container and boil it for 30 minutes, then put the water in it. We originally had thought of using a gallon glass jar, but with so many plants I think we will switch to a 5-gallon container which is completely enclosed and has a spigot on one side. I'll try and work out with our friend a proper way to sterlize this container. It was specifically made for containing potable water for camping etc. so I think there is a way to do this. As far as I can tell, he can pray for the water from 70 miles away. He has not indicated that he needs to see the water, or come in contact with it. When we first moved here I learned the procedure considered correct for getting a clean water sample out of the tap: using a match on the spigot for a set number of minutes, letting the water run for a set number of minutes, then taking the sample; and I will use this procedure to fill his container of prayer water, so as to assure it has not been contaminated. (We have a well. There is no concern about chlorine or other additions.)

I like the idea of either picking the numbers out of a hat, or simply flipping a coin. With the pot being in either group A or group B the coin flipping should work just fine, so I think I'll use the coin flipping.

I'll be running the experiment because I don't come in contact with this guy; he works with Mr. Amapola and is more his friend than mine. I only see the guy very occasionally, and not up here where we live. So for this reason, I won't tell Mr. Amapola to which group each pot belongs.

I'm sorry to hear I don't have enough pots to make it a proper experiment. Perhaps I'll try and design another experiment - but *NOT* one where I have to measure blades of wheat grass!

 

[69dodge]

I still think it's the binomial distribution though as the probability of guessing right on any of the 20 trials is still .50.

Yes, each guess individually has probability 1/2 of being correct, but this is not enough to ensure that the number of correct guesses is distributed binomially. For that, the guesses also need to be independent, and here they aren't.

(Consider an extreme example: A single coin is secretly flipped, to determine whether pots 1-10 or 11-20 are prayed for. The guesser, knowing this, will then guess either that pots 1-10 were prayed for, or that pots 11-20 were. Here too, each pot has probability 1/2 of being guessed correctly. But the number of correct guesses is definitely either 0 or 20, which is not binomial at all. Guessing all 20 pots correctly wouldn't be significant, because doing so has the very large probability of 1/2.)

But I suppose it doesn't make that much of a difference here, practically speaking. Though we disagree about the exact p-values, we agree that 16 correct guesses are enough, whereas 14 are not.

 

[Jeff Corey]

...I like the idea of either picking the numbers out of a hat, or simply flipping a coin. With the pot being in either group A or group B the coin flipping should work just fine, so I think I'll use the coin flipping...

I'd strongly advise that you pick the numbers out of a hat or use some other method of sampling without replacement. This would insure you have ten plants in each group and be able to set p = .5 in the binomial test.
In class experiments with 20 subjects, I use a deck of 20 cards, 10 red and 10 black suits, shuffle them and assign conditions based on which color they draw.

 

[69dodge]

Actually we are performing the experiment at his insistence, because we (Mr. Amapola and I) do *NOT* believe in prayer and he does. He is convinced that we will see the light as this experiment progresses. [...] I'll be running the experiment because I don't come in contact with this guy; he works with Mr. Amapola and is more his friend than mine. I only see the guy very occasionally, and not up here where we live. So for this reason, I won't tell Mr. Amapola to which group each pot belongs.

Have you thought about how exactly it will be decided, at the end of the experiment, whether prayer in fact helped the plants it was applied to?

Or is the guy just assuming that the difference between the two groups of plants will be so large as to be completely obvious to you?

 

[fls]

Thanks for clarifying the situation.

Do you really need to go through the process of blinding and randomization then? If he is not involved in viewing the experiment, you don't have to prove anything to him. And I agree that two groups of 10 are fine if it's just you looking at the plants (that extra layer of uncertainty is not necessary).

Linda

 

[Jeff Corey]

Have you thought about how exactly it will be decided, at the end of the experiment, whether prayer in fact helped the plants it was applied to?

Or is the guy just assuming that the difference between the two groups of plants will be so large as to be completely obvious to you?

One objective measure would be to weight the plants (but not the way the cops weigh pot plants, roots, dirt and all).

 

[bpesta22]

Not sure if you intend on doing this, but if you boil the water or do anything else to it besides pray, then the control group water should also be boiled, etc.

Start with a big drum that contains the water for the whole experiment (after you've boiled it filtered it or whatever you're doing) then mix into two containers right before you start watering and let the prayers begin.


Dodge, I still respectfully disagree. It's not the guesses that matter when it comes to the independence assumption, it's the probability of being correct on each trial, which does not change.

 

[Amapola]

He's assuming there will be a really huge difference - that it will be a "better" plant than normal. Somewhere around here I have a link he gave me to some goofy website that talks about praying for water and how the prayer supposedly changes the structure of the water. I'll look for it and post it when I find it.

I agree though - we need to devise a way to show the praying worked. I just thought that if the guy comes out here and looks at the 20 pots and can not tell the difference between them, that would demonstrate it did not work. I see now I should ask him how I'll know the difference.

Thanks Dr. Corey - that sounds like a better idea, I'll get 20 cards with 10 black, 10 read, shuffle them, and then just walk along the row of pots and turn up one card at a time, and assign the pot I'm next to to that group. You are right, if I use the coin flipping, I may not end up with exactly 10 in each group.

 

[bpesta22]

You could use coin flipping and when you get to 10 in any group, the rest would be of the other group. Just as random

This would be a good case study in experimental design. The idea is totally simple, but implementing it to fairly test the idea is not.

 

[Amapola]

Thanks for clarifying the situation.

Do you really need to go through the process of blinding and randomization then? If he is not involved in viewing the experiment, you don't have to prove anything to him. And I agree that two groups of 10 are fine if it's just you looking at the plants (that extra layer of uncertainty is not necessary).

Linda

The reason I am doing the randomizing is this: I am assuming I will *NOT* be able to discern any difference whatsoever. And once we tell our friend that, I am also fairly certain he will want to come up and see for himself. So I don't want all the prayed for ones right next to each other, or all in a clump. Also, I want to be certain the plants have a "fair" chance, so I want them to be randomized in the row in case one end has a problem or something.

Bpesta22, we're not going to boil the water... we were going to boil the container, in order to avoid contamination. After that it would just be the water that comes out of our well. I think it's OK, because we are not measuring against anything other than "normal" water, and the normal water that has been prayed for, if that makes any sense...

I should also say I will use two identical containers (a pint each, I think) and give all the pots the same amount of water each day, one container being designated for the normal water and one for the prayed for water.

I think I will take pictures as I go along. Perhaps I can get the local paper to print up the story afterwards...

 

[Amapola]

You could use coin flipping and when you get to 10 in any group, the rest would be of the other group. Just as random

This would be a good case study in experimental design. The idea is totally simple, but implementing it to fairly test the idea is not.

I was just thinking this very thing...

 

[CFLarsen]

There is no real concern about the plants being messed with (except that *Claus* has already started praying! )

Ask your friend how he would control for that.

 

[Amapola]

Hey, I found that link I was talking about: linky-poo.

Maybe some of you can make better sense of this link than I can. They photographed water as it crystallized and the crystals were all different. Isn't that the way it is supposed to occur? I thought I learned as a kid that snowflakes don't all look alike... am I wrong here?

Claus, I will ask him if other people praying will mess things up, and warn him that some mad Dane might be praying for who knows what...

Of course the whole thing is remarkably silly, but this guy is a very good friend of Mr. Amapola, and a very earnest and nice person, so we are taking him as seriously as we can. I am doing my best to honestly set up a good experiment that will have results (one way or the other!) that were not arrived at by cheating or faulty methodology, as far as it is possible within my means. I appreciate all the suggestions. You guys have covered things I did not think about that should be thought of. Even controlling for *other* people praying ought to be taken into consideration.

 

[ben m]

Let me propose a blinding method which would solve most of the problems, including judgement and tampering issues.

(1) Label each plant with a letter A-T.
(2) Label 20 paper cups with the letters A-T.
(3) In secret, a trusted 3rd party (Person C) should randomly draw 10 cups to be "prayed for"; this should be written down and kept by Person C but not shown to anyone.
(4) Every watering-period, the experimenters provide Person C with a bucket of holy water and a bucket of regular water. In secret, person C fills each cup with an appropriate type of water, then leaves the room.
(5) The experimenters return and water each plant from its designated cup.
(6) The experimenters randomly shuffle the plants on their shelf so that crowding/light differences don't have too large an effect.
(7) Repeat steps 4-5 the agreed-upon number of times.
(8) At the end of the experiment, the experimenters "rate" each plant on whatever scales they like (Height? Mass? Number of leaves?), or subjectively rank the plants to identify the 10 best.
(9) Person C returns and reveals which plants had gotten holy water. This list is compared with the best-plants list determined in Step 8.

In this case, even a maximally-biased, cheating experimenter can't change the results; they don't know which plants to discreetly kill or fertilize or shadow, nor which ones to overrate in the evaluation step.

But it's more important to agree ahead of time on how stats work; you don't want him or her to guess 12/20 right and start claiming victory.

 

[Jekyll]

You could use coin flipping and when you get to 10 in any group, the rest would be of the other group. Just as random

This would be a good case study in experimental design. The idea is totally simple, but implementing it to fairly test the idea is not.

I would probably split the plants into ten pairs and give one of each pair holy water and the other unholy.
You can assign the splitting of the pairs by coin flip.

Guessing the pairs correctly will then have a symmetric binomial distribution of the size 10 and is nice and tractable.

 

[69dodge]

I'd strongly advise that you pick the numbers out of a hat or use some other method of sampling without replacement. This would insure you have ten plants in each group and be able to set p = .5 in the binomial test.

I agree about sampling without replacement. (The link I gave in post #10 is to a webpage that can be used to do that automatically, without a hat. It lists the numbers from 1 to 20 in random order. Pray for the first ten listed, don't pray for the last ten.)

You agree with bpesta22 that a binomial test is appropriate? I still disagree. Can you explain your reasoning? Where are the independent Bernoulli trials?

Not sure if you intend on doing this, but if you boil the water or do anything else to it besides pray, then the control group water should also be boiled, etc.

Start with a big drum that contains the water for the whole experiment (after you've boiled it filtered it or whatever you're doing) then mix into two containers right before you start watering and let the prayers begin.

Right, that makes sense. Everything should be the same, except that half the water gets prayed for and half doesn't.

Dodge, I still respectfully disagree. It's not the guesses that matter when it comes to the independence assumption, it's the probability of being correct on each trial, which does not change.

Are we talking about the same setup?

There are twenty plants. Ten of them were given prayed-for water. Ten were given regular water. The guy knows this; he just doesn't know which are which. He will try to decide which are which.

There aren't really twenty separate trials. He has one big job to do: separate the twenty plants into two groups of ten each.

If we think of his guessing the status of each plant as a separate trial, the probability of being correct on the next trial does change, depending on whether the previous trials were correct. For example, if he guesses that the first ten plants were prayed for, and he is told that those guesses were correct, he is sure to guess correctly about the remaining ten plants. The last ten trials no longer have their original probability (1/2) of being correct.

I agree though - we need to devise a way to show the praying worked. I just thought that if the guy comes out here and looks at the 20 pots and can not tell the difference between them, that would demonstrate it did not work. I see now I should ask him how I'll know the difference.

No, that's ok. You said he lived far away, so I wasn't sure what the plan was. If he can look at the plants himself, that's the best. You can let him use whatever method he wants, to pick out the prayed-for plants.

But you and he should try to agree beforehand on how many correct picks are "enough". Even if prayer doesn't affect the plants at all, so that in effect he'll simply be guessing, he'll get a few right anyway, just by chance.

Does he think he'll be able to make at least 16 correct identifications out of 20? That is, out of the 10 plants he says were prayed for, he's allowed one or two mistakes, but no more?

I would probably split the plants into ten pairs and give one of each pair holy water and the other unholy.
You can assign the splitting of the pairs by coin flip.

Guessing the pairs correctly will then have a symmetric binomial distribution of the size 10 and is nice and tractable.

That would work. But only one mistake would be allowed then, rather than two, for the same level of significance (about 1%).

 

[Jekyll]

That would work. But only one mistake would be allowed then, rather than two, for the same level of significance (about 1%).

Sure, and I agree with your earlier post (which I missed the first time around) about the distribution being:

[latex]$\frac{\binom{10}{n/2}\times\binom{10}{10-n/2}}{ \binom{20}{10}}$[/latex]
for the probability of correctly labelling an even number n, of plants in the original partitioning problem.

But it seems to me, there's a direct trade off between the power of the test, and how many people can follow the maths.

 

[Jeff Corey]

I agree about sampling without replacement. (The link I gave in post #10 is to a webpage that can be used to do that automatically, without a hat. It lists the numbers from 1 to 20 in random order. Pray for the first ten listed, don't pray for the last ten.)

You agree with bpesta22 that a binomial test is appropriate? I still disagree. Can you explain your reasoning? Where are the independent Bernoulli trials?



Right, that makes sense. Everything should be the same, except that half the water gets prayed for and half doesn't.



Are we talking about the same setup?

There are twenty plants. Ten of them were given prayed-for water. Ten were given regular water. The guy knows this; he just doesn't know which are which. He will try to decide which are which.

There aren't really twenty separate trials. He has one big job to do: separate the twenty plants into two groups of ten each.

If we think of his guessing the status of each plant as a separate trial, the probability of being correct on the next trial does change, depending on whether the previous trials were correct. For example, if he guesses that the first ten plants were prayed for, and he is told that those guesses were correct, he is sure to guess correctly about the remaining ten plants. The last ten trials no longer have their original probability (1/2) of being correct...

I am assuming that he will make all his guesses in one sitting and not be given feedback until all guesses are recorded. This would make each guess independent of the previous ones, especially if he were not told how many were in each group.
So a Binomial or Sign test should be appropriate.

 

[Amapola]

The way Dr. Corey describes is the way I had planned to do it - let the guy walk around and look at all the plants, and say one by one which plant belongs in which group. When he was finished, then I would produce the map that showed which plants were given the prayed for water and which the normal water. With only me knowing which were which, I would probably have my husband go around with him and record his answers.

69dodge, I'll check out that link! That's a nice high-tech way to get the numbers.

The more I think about this the more convinced I become that the guy will drop out and refuse to say which plants belong in which group before the end. Maybe I need more faith in his faith.