Case Study: The IIG Protocol for VFF

[GeeMack]

The protocol of my alternative test suggestion has nothing to do with the IIG protocol other than being an alterative protocol. Make sense now?

You want to know “what they did wrong” but you don’t like me saying what I think they did wrong and offering an alternative. Then you say you want to know “what other things they could or could not have done” but when I offer such other things you insult me. This is how you conduct a study?

Oh, so what they did wrong was to test her for the actual claim she made, and you think it would have been better to test her for a claim she didn't make? This is how you conduct a study?

I’ve said many times that I would prefer her to identify people not kidneys. I would not run an actual test for her that required her to identify kidneys. The test I’m suggesting is purely hypothetical.

Since the number of claims she didn't make is one less than infinity, why do you think testing her for detecting people missing kidneys is any better than testing her for lifting stones with psychokinesis? They are both claims she didn't make, and obviously her actual claim doesn't mean anything in your reality. After all, it would be much more interesting to see someone lift (or fail to lift) a stone with psychokinesis, wouldn't it?

To say that a person is missing a kidney from the right side is absolutely saying that the person is missing a kidney. If you don’t agree please explain exactly why.

To say a person is missing a kidney is not exactly the same as saying that a particular kidney is missing.

If she sees a person's body part she sees part of the body of a person. It's reasonable to assume that the rest of the body is there as well. Let me change the test to “Can you see a body part of a person? Doesn’t have to be a kidney.” Is that better?

Not better if you're trying in any way to stay on topic with the conversation in this thread. How about you hold your thoughts until someone makes that claim, then take your comments to that discussion? For some unapparent reason you keep trying to shove your fantasy claim and claimant into this discussion, and it's making you look like a troll.

 

[Uncayimmy]

First off, thanks for addressing my questions and points directly. I know it should be standard practice, but others <cough ynot cough> could learn from this exchange.

Never considered that case. Was thinking more along the lines of a random sampling of 10. It could work out all the available 1 kidney people end up in a single round.

What is the advantage of that approach versus fixing it at one target per round? Is it simply to decrease the odds of total success by luck? Are you sticking with three targets total and telling the claimant as much?

One thing I don't like is that nobody knows the odds of any given guess until after the test. The other issue I have is that not all guesses have an equal likelihood of being right, so while the odds of success by guessing remain the same, the value of a successful guess might not be meaningful. I'm going to research this issue more.

Increases randomness. People inherently look for patterns, but they also create them. It significantly decreases her probability of succes per round. Less fidgeting as well.

If I didn't like you, I'd nitpick over the notion of how randomness can be increased or decreased. It's like being more or less pregnant.

Are you saying that 1 in 1,728 was not sufficient?

Staging. Have people behind a wall or screen of whatever, so that you simply draw the curtain when she's ready. It's fairly quiet, they walk out, sit down and ready to go.

As I think more about this, the entrance time can be discarded. See below.

True, but I think a single submission per person, all final (it's not like they're going to grow a kidney or take one out, so final is final) eliminates any disputes.

You have to protect against the IIG being accused of cheating. The easiest way for them to cheat is to switch numbers off-stage. Therefore, if you require a decision on each person, you have to do the routine with the unveiling and taking of pictures. Otherwise the results are not self-evident when the sonogram testing is done. I think this alone is the biggest drawback - it's time and labor intensive.

I'd say it could be done in about 4 hours. I've had 2 exams in one day, no excuses.

You're not an irrational claimant spending $1K on a one-time shot at $50k being broadcast live and covered in the press.

Well there's the catch. You can't make her agree, but I think this is realistic. After round 1 you ultrasound and get results. If she flunks out, she flunks out. If she wishes to continue for research purposes I suppose you entertain her.

Early stopping is not an option as far as I'm concerned. First, if she's right, you don't stop. This gives her feedback that whatever she might be doing is working. More importantly, suppose upon reviewing the video a protocol violation or other mistake by the IIG is noticed. If you stopped after the first round, you have to give a retest. If you didn't stop and the other two rounds were handled without any problems, then a failure in either of those rounds will be sufficient grounds to squash any demands for a retest.

These are two good reasons why the IIG would not suggest early stopping and refuse any request to do so. The only advantage to doing it is to save some time on the testing day.

 

[Uncayimmy]

Thought we were discussing the protocols of testing not whether we should make a test to suit the demands of anyone.

How many times do we have to explain that the challenge protocol is negotiated and not dictated? Maybe you also need to be reminded that it's a publicity stunt with a signed contract and not a scientific test. It's run by volunteers with limited resources, so time, money, materials, and volunteers are limited. It's also a one-shot deal, so you have to be very confident that everything will go off without a hitch.

If you would make your suggestions and then address these issues yourself, you could save everyone a lot of time. It's getting really old having to do it for you.

Why is a purely intellectual exercise wasting time? I'm trying to offer suggestions toward any future testing done by anyone (that will get a laugh from UY).

No, you're not talking about "any" test for "any" person. You might think you are, but you're not. In general we're talking about detecting the absence of a kidney, which means an extremely limited number of people. In fact, as best as I can determine looking at the JREF and IIG websites, Anita is the only one ever to present such a claim for testing to these two organizations.

And I will remind you once again to read the subject line. All suggestions need to be evaluated in terms of whether VFF would have been likely to accept them or not. You are welcome to argue that maybe somebody else might be okay with it, but we still have to discuss whether Anita would have accepted it.

 

[Furcifer]

What is the advantage of that approach versus fixing it at one target per round? Is it simply to decrease the odds of total success by luck? Are you sticking with three targets total and telling the claimant as much?

You've answered your own question. She knows for a fact at least 1 person in 6 is missing a kidney. You've given her information she already claims to know. This allows her to differentiate the subjects and pick up on possible signals. The odd ball in the group, IMHO is more likely to be the target than the others. That is if I know I have to pick at least one. The odd ball is less odd if I'm not sure anyone in the group is actually missing a kidney. The one at a time scenario also makes it harder to make a relative comparison. Not impossible, but certainly harder to do.

One thing I don't like is that nobody knows the odds of any given guess until after the test. The other issue I have is that not all guesses have an equal likelihood of being right, so while the odds of success by guessing remain the same, the value of a successful guess might not be meaningful. I'm going to research this issue more.

I think the odds on a single guess would depend on the guess itself. A (L) or (R) guess would have a significantly lower probability than a (Both). Each guess would depend on the number of 1 kidney subjects in the group or round.

If I didn't like you, I'd nitpick over the notion of how randomness can be increased or decreased. It's like being more or less pregnant.

lol, feel free to nit pick. I think in this case it's a proper term because it's a relative comparison. Feel free to ask a woman in her 3rd trimester who's more pregnant, her or her friend who just peed on a stick and found out.

Are you saying that 1 in 1,728 was not sufficient?

No, overall probability was fine, but I was kinda shocked to see that the probability per round was 1 in 12. Even with 5 people per round as I have outlined, I think the overall probability per round is 1 in 243.

You have to protect against the IIG being accused of cheating. The easiest way for them to cheat is to switch numbers off-stage. Therefore, if you require a decision on each person, you have to do the routine with the unveiling and taking of pictures. Otherwise the results are not self-evident when the sonogram testing is done. I think this alone is the biggest drawback - it's time and labor intensive.

Smile for the camera. All you need is a picture. Filming will also prevent this from happening. I say it's a non-issue in this day and age.

Early stopping is not an option as far as I'm concerned. First, if she's right, you don't stop. This gives her feedback that whatever she might be doing is working. More importantly, suppose upon reviewing the video a protocol violation or other mistake by the IIG is noticed. If you stopped after the first round, you have to give a retest. If you didn't stop and the other two rounds were handled without any problems, then a failure in either of those rounds will be sufficient grounds to squash any demands for a retest.
These are two good reasons why the IIG would not suggest early stopping and refuse any request to do so. The only advantage to doing it is to save some time on the testing day.

Yes, but I only added that because in this scenario the odds are so low. It's not necessary. In fact early stopping would totally ruin my next point below.

I think the added benefit is you can adapt on the fly. Even if you found 4 protocol violations in a single round, you end up with an unweighted probability per round of 1 in 729. Now you've got that going for you, plus you have 2 more rounds which essentially amount to tests in and of themselves.

I've been trying to crunch the numbers in my head, but I simply don't recall enough stats. I'm not entirely sure the odds I calculated are even correct. I'm pretty sure but not positive.

I will say this, when you really sit down and think about it, the IIG did a pretty good job with the resources and time allotted. It seems to me they also limited the damage of a completely random correct guess. In my test, a lucky guess would carry much more weight. That could be a problem.

 

[Uncayimmy]

No, overall probability was fine, but I was kinda shocked to see that the probability per round was 1 in 12. Even with 5 people per round as I have outlined, I think the overall probability per round is 1 in 243.

I'll comment more on this later. I need to check some stuff. Didn't want you to think I was being disrespectful by asking you questions and blowing off your responses. Not that anyone in this thread does that...

Smile for the camera. All you need is a picture. Filming will also prevent this from happening. I say it's a non-issue in this day and age.

What they did at the end of each trial (with the exception of the group photo) would be what they would do in your scenario. I say they have to wait until the photo is printed so that she can sign and accept her choice before moving on. From a technology standpoint, it's no big deal, but it's still going to be at least two minutes. So, my calculation says 7 minutes per person minimum. What do you estimate?

I'll reply to your other points later. Gotta run.

 

[wardenclyffe]

Keep in mind how quickly she gets tired. Also the break times. I assume the breaks between trials in the IIG protocol were something that she wanted. With a one at a time test, who knows how many breaks (and for how long) she'll want?

Ward

 

[ynot]

Keep in mind how quickly she gets tired. Also the break times. I assume the breaks between trials in the IIG protocol were something that she wanted. With a one at a time test, who knows how many breaks (and for how long) she'll want?

Ward

A “one at a time” series of consecutive tests that stops as soon as an incorrect answer is given would have half a chance of only taking the time of the first test (4.5 - 5 minutes). Three times more likely to fail than succeed in the second (9 - 10 minutes), seven times more likely to fail than succeed in the third (13.5 - 15 minutes), fifteen times more likely to fail than succeed in the forth (18 - 20 minutes), etc. On the odds of lucky guess a lot shorter time than the IIG test would be expected. The total number of test people that would be expected to be required would also be less.

 

[Furcifer]

What they did at the end of each trial (with the exception of the group photo) would be what they would do in your scenario. I say they have to wait until the photo is printed so that she can sign and accept her choice before moving on. From a technology standpoint, it's no big deal, but it's still going to be at least two minutes. So, my calculation says 7 minutes per person minimum. What do you estimate?

I just want to make one thing clear to anyone reading, especially VFF, I don't think she would have cheated or claimed the IIG cheated. She may have a tons of excuses but she never cried foul. By talking about insulating the IIG from accusations I don't want to insinuate this was ever an issue.

That being said, it has to be a consideration when designing the protocol.

I'm not certain a photo has to be printed. You snap a shot of the paper, you have a judge take it over to the subject and you snap a shot. 2 pics, 30 seconds tops.

 

[Furcifer]

Keep in mind how quickly she gets tired. Also the break times. I assume the breaks between trials in the IIG protocol were something that she wanted. With a one at a time test, who knows how many breaks (and for how long) she'll want?

Ward

That's why a couple of minutes to take a picture and set up might not be a bad thing. A little rest in between.

There's no way to prevent someone from making up a story about fatigue. I think the short bursts with a pause make the claim less valid than the long grueling 45 minute rounds. Oh the humanity!

 

[Furcifer]

A “one at a time” series of consecutive tests that stops as soon as an incorrect answer is given would have half a chance of only taking the time of the first test (4.5 - 5 minutes). Three times more likely to fail than succeed in the second (9 - 10 minutes), seven times more likely to fail than succeed in the third (13.5 - 15 minutes), fifteen times more likely to fail than succeed in the forth (18 - 20 minutes), etc. On the odds of lucky guess a lot shorter time than the IIG test would be expected. The total number of test people that would be expected to be required would also be less.

Do you remember the Mike Tyson/Peter McNeely fight? Probably not, it only lasted 89 seconds. It pissed a lot of people off. People paid good money to watch a fight that didn't really happen. It was considered a mockery.

That's the problem I see with your protocol. You run the risk of pissing people off. Not only that, but it's probably going to disuade anyone from ever coming forward and submitting to the challenge.

In your protocol, how many people are there in total and how many are missing a kidney?

 

[ynot]

Do you remember the Mike Tyson/Peter McNeely fight? Probably not, it only lasted 89 seconds. It pissed a lot of people off. People paid good money to watch a fight that didn't really happen. It was considered a mockery.

That's the problem I see with your protocol. You run the risk of pissing people off. Not only that, but it's probably going to disuade anyone from ever coming forward and submitting to the challenge.

In your protocol, how many people are there in total and how many are missing a kidney?

It’s only a problem if you want to entertain rather than establish the truth. Taking too long also pisses people off (as the IIG test did).

Don’t want to keep discussing my hypothetical test because it pisses people off.

 

[Uncayimmy]

A “one at a time” series of consecutive tests that stops as soon as an incorrect answer is given would have half a chance of only taking the time of the first test (4.5 - 5 minutes). Three times more likely to fail than succeed in the second (9 - 10 minutes), seven times more likely to fail than succeed in the third (13.5 - 15 minutes), fifteen times more likely to fail than succeed in the forth (18 - 20 minutes), etc. On the odds of lucky guess a lot shorter time than the IIG test would be expected. The total number of test people that would be expected to be required would also be less.

Yeh, we get it. Stopping early saves time. It might save none, some, or a lot - thanks for explaining that. I didn't realize that was a goal. We have the goal of setting a maximum time that is acceptable to all parties in involved, but beyond that why are you so concerned with letting people go early?

Also, why don't you address my two concerns about stopping early and weigh the pros and cons. I'll start.

Pros of stopping early
* People might get to go home early.

Now, you finish.

 

[Furcifer]

It’s only a problem if you want to entertain rather than establish the truth. Taking too long also pisses people off (as the IIG test did).

Don’t want to keep discussing my hypothetical test because it pisses people off.

I don't think anyone is pissed off. I think you may have glossed over some things attempting to make your point. It's frustrating trying to engage someone in a discussion when they do this. To avoid any confusion try a more systematic approach.

 

[Uncayimmy]

That's why a couple of minutes to take a picture and set up might not be a bad thing. A little rest in between.

There's no way to prevent someone from making up a story about fatigue. I think the short bursts with a pause make the claim less valid than the long grueling 45 minute rounds. Oh the humanity!

I'm not concerned with people making up stories about fatigue after the fact. They can and will say anything they want after failing. Short of a non-disclosure agreement, there's not much you can do about it.

I'm concerned about getting a claimant to agree to a maximum time. Sometimes I think people don't appreciate the stress of being on a stage under hot lights and performing in front of a crowd that expects you to fail. It's quite taxing. It's especially stressful because any mistake might mean failure (will, in this case).

I have no problems with somebody not wanting to exceed 45 minutes to an hour tops. When I play a gig, that's about my limit per set. And I'll tell you this: the times between songs don't count as mini breaks.

 

[Furcifer]

I'm not concerned with people making up stories about fatigue after the fact. They can and will say anything they want after failing. Short of a non-disclosure agreement, there's not much you can do about it.

Yah, it's funny you mentioned that. I was going to say before that's basically the only way you're ever going to get anybody to shut up afterwords, a gag order. I termed it the "Shut up and go away" clause.

I'm concerned about getting a claimant to agree to a maximum time. Sometimes I think people don't appreciate the stress of being on a stage under hot lights and performing in front of a crowd that expects you to fail. It's quite taxing. It's especially stressful because any mistake might mean failure (will, in this case).

I assume this is because you're dealing with volunteers. It's too bad there's no real money involved here, or you could actually pay people to do this. $200 bucks to sit still for 8 hours isn't that bad. As for the stress, I totally agree. I thought it was a cop out on Anita's part, but if you consider what's really going on here it's very taxing. They're in a fight for their paranormal lives. I've herd they pipe in soothing music at slaughterhouses to ease they animals down the chute. I hate to be blunt but I think this amounts to the same thing. There's no reason to crush the claimant, you want them to be comfortable and accept what's happening around them. If you approach this in this manner there's a much greater chance afterwords the claimant will be more skeptical of their own claims and less defensive.

I have no problems with somebody not wanting to exceed 45 minutes to an hour tops. When I play a gig, that's about my limit per set. And I'll tell you this: the times between songs don't count as mini breaks.

As long as they (the claimant) doesn't dictate to an extent the odds are in their favour. I'd say the 1 in 2000 overall odds was a good way to encourage Anita's participation without accidentally lending credibility to her claim in the case of a lucky "hit". At the same time, I think the 3 in 12 overall odds of her making a "hit" played a little too much into her hands. Not for me or you, or the folks at IIG, but the fence sitters. If they're are sitting on the fence they are already giving too much leeway. An odds on result may not convince them. I'd personally prefer an overall "hit" at around 50 to 1.

And unless you're the drummer, suck it up

 

[ynot]

As the IIG test had a mutually agreed protocol of 100% success, not stopping as soon there was a failure didn’t alter the overall outcome. It did however alter the perceived outcome to many people (not just “true believers“). It really doesn’t matter to these people what the protocol says. In their mind it was a 100% success and a 50% success against a 100% failure. This is what I didn’t like about the test not stopping with the first failure. It gave her an unnecessary chance to get lucky and she did.

As I understand it the main reason for not having a protocol for stopping with the first failure was because if it did every answer that didn’t stop the test was obviously a correct answer. And that this would have given her an advantage of knowing that any non-parnormal method she may have been using would have been working.

If half a dozen or more correct answers could have been given this may have been so but in the IIG test there was only ever going to be a maximum of three. Obviously the first answer couldn’t be “advantaged” so this only left two answers that coluld have been (2nd and 3rd).

If she had a non-paranormal method of identifying the correct person that she could be certain was more likely the cause of the success than lucky guess then she would have used that method regardless of knowing that the previous answer was correct.

It’s also highly unlikely that there would have been such a method that would have been present and evident in all three correct people. If there was why wouldn’t one of the many sceptics watching have noticed it? There’s no evidence that a non-paranormal method was possible or was being used.

Given this was only a preliminary test and a “publicity stunt” I think any slight advantage she may have gained from knowing her previous answer was right was less of a risk than the risk of giving her more unnecessary chances to make lucky guesses (as she did).

 

[Uncayimmy]

With regards to the time factor, we have been concentrating on getting the claimant to agree. We also have to think about the volunteer subjects. They have to stay sequestered for the duration of the test, which is really the worst part for them. It's extremely boring to be locked up in a room for several hours. It's tough enough as it is to find people. It's not like you can really go back and change the protocol if you find out you can't find enough volunteers.

Another important consideration is that you only get one shot. If you screw up, you might end up having to do a retest at your own expense, which would really suck. Every step you take is an opportunity for something to go wrong. Contrary to ynot's assertions, you're not going to get a full dress rehearsal. So it's a good idea to keep it as simple as possible.

I know, how hard is it to bring in a person, have them sit, then have the claimant fill out a form, take a picture, print it, attach it to the form, sign it, file it, and remove the person? It's not rocket surgery, but the fact remains that doing this 30 time greatly increases the risk of a stupid mistake or unforeseen accident compromising the test. The last thing you want is for a dispute to arise, even if it's minor.

What was good about the IIG protocol is that it made the required number of perfectly executed steps about as small as possible while still maintaining very difficult odds. Let's say you are 99% sure can execute the whole swapping and documentation routine properly (a fair estimate considering we're talking about humans and computers in a strange environment). If you do it 30 times, there's a 26% chance you'll make a mistake. If you do it three times, there's a 3% chance.

In the end it's a judgment call. You have to balance the pros and cons. I'm not sold that the benefits outweigh the risks.

 

[Uncayimmy]

As I understand it the main reason for not having a protocol for stopping with the first failure was because if it did every answer that didn’t stop the test was obviously a correct answer.

That's not the main reason in my book. The main reason is the one you're not addressing, which is that if there is a problem in a trial that does not affect the other trials, then it's much more difficult for the claimant to demand a retest. In the worst case scenario she fails the first trial and upon reviewing the video she finds a protocol violation that worked against her. She then demands a retest or a refund.

Please address that concern.

If half a dozen or more correct answers could have been given this may have been so but in the IIG test there was only ever going to be a maximum of three. Obviously the first answer couldn’t be “advantaged” so this only left two answers that coluld have been (2nd and 3rd).

Yes. I said this before. It's a *huge* problem in your suggestion to use only two subjects. It's much less of a problem in the IIG protocol.

If she had a non-paranormal method of identifying the correct person that she could be certain was more likely the cause of the success than lucky guess then she would have used that method regardless of knowing that the previous answer was correct.

You make too many assumptions. It's not just the possibility of her having a technique developed in advance that she's sure will work. Hell, it could be as simple as somebody offering to split the $50K by coughing the right number of times as a signal. She hears two sets of coughs and has to choose one she thinks is the signal. Maybe she thinks the person is lying and just trying to set her up by giving her the wrong answer.

If you don't tell her she's correct, then she still has the element of doubt.

Given this was only a preliminary test and a “publicity stunt” I think any slight advantage she may have gained from knowing her previous answer was right was less of a risk than the risk of giving her more unnecessary chances to make lucky guesses (as she did).

So, now you're saying the reason to stop early is to keep her from making "extra" guesses and getting lucky?

 

[tsig]

As the IIG test had a mutually agreed protocol of 100% success, not stopping as soon there was a failure didn’t alter the overall outcome. It did however alter the perceived outcome to many people (not just “true believers“). It really doesn’t matter to these people what the protocol says. In their mind it was a 100% success and a 50% success against a 100% failure. This is what I didn’t like about the test not stopping with the first failure. It gave her an unnecessary chance to get lucky and she did.

As I understand it the main reason for not having a protocol for stopping with the first failure was because if it did every answer that didn’t stop the test was obviously a correct answer. And that this would have given her an advantage of knowing that any non-parnormal method she may have been using would have been working.

If half a dozen or more correct answers could have been given this may have been so but in the IIG test there was only ever going to be a maximum of three. Obviously the first answer couldn’t be “advantaged” so this only left two answers that coluld have been (2nd and 3rd).

If she had a non-paranormal method of identifying the correct person that she could be certain was more likely the cause of the success than lucky guess then she would have used that method regardless of knowing that the previous answer was correct.

It’s also highly unlikely that there would have been such a method that would have been present and evident in all three correct people. If there was why wouldn’t one of the many sceptics watching have noticed it? There’s no evidence that a non-paranormal method was possible or was being used.

Given this was only a preliminary test and a “publicity stunt” I think any slight advantage she may have gained from knowing her previous answer was right was less of a risk than the risk of giving her more unnecessary chances to make lucky guesses (as she did).

If they stopped after one test you would be here arguing they should have carried out them all.

 

[Furcifer]

In the end it's a judgment call. You have to balance the pros and cons. I'm not sold that the benefits outweigh the risks.

That's true, and I'm not a salesman. My personal opinion is that you reduce the chance for a cold read by decreasing the amount of time the claimant has to make comparative evaluations. If that means increasing the occurrences for which the claimant can make accusations about cheating so be it. I say it's easier to deal with those claims than explaining why they were able to make a cold read and get a "hit".

With regards to this recent success, I think the IIG should be considering what kind of preliminary odds they need to give in order to encourage more challenge participants. Perhaps halving the prize to $25K and increasing the odds to 200 to 1 may be in order.