Meta-analysis of JREF tests

[T'ai Chi]

What does everyone think about this:

Would it be desirable to perform a meta-analysis(or analyses) of all JREF tests that were conduced in a similar (ideally exact) manner?

Yes/no? Because... ?

What do people think the result would be?

 

[Mercutio]

A. each test could be different, as the procedure depends on the claims of the claimant. So while some may be similar, I would not like to see the flexibility of the challenge subsumed to some desire for meta-analysis.

B. unless I am very much mistaken, thus far no claimant has passed even the preliminary test. Speaking as a stats teacher, I tell you there are times when no statistics are needed.

 

[EdipisReks]

that would be one incredibly easy and quick meta-analysis.

 

[T'ai Chi]

A. each test could be different, as the procedure depends on the claims of the claimant. So while some may be similar, I would not like to see the flexibility of the challenge subsumed to some desire for meta-analysis.

B. unless I am very much mistaken, thus far no claimant has passed even the preliminary test. Speaking as a stats teacher, I tell you there are times when no statistics are needed.

Um, it doesn't matter if someone hasn't passed even a preliminary test. You can do a meta-analysis on the statistics still.

 

[EdipisReks]

Um, it doesn't matter if someone hasn't passed even a preliminary test. You can do a meta-analysis on the statistics still.

why bother?

 

[Darat]

What does everyone think about this:

Would it be desirable to perform a meta-analysis(or analyses) of all JREF tests that were conduced in a similar (ideally exact) manner?

Yes/no? Because... ?

What do people think the result would be?

Considering that you believe that science can only be done in a laboratory I suspect you have an agenda with this question rather than wanting a discussion, especially since you've said in the past such comments as

Originally posted by T'ai Chi here

SRW, Randi is fine for the most part, it is his followers that are dogmatic.

Good results are obtained from science, and science is done in peer reviewed journals. Interesting entertainment, such as the challenge, is entertaining and thought provoking, but ultimately doesn't show much in the long run.

And

Originally posted by T'ai Chi here


…snip…

Science is done in labs and written about in journals.

…snip…

However assuming that you are now of the opinion that the challenge is “science” and has the controls necessary for the results to be taken “seriously” I still can’t see how a meta-analysis could be done on the Challenge. Doesn’t a meta-analysis assume there is some “commonality” in the experiments and resulting data? How would you go about performing a meta-analysis on the results which are simple “fail” or “passes”?

 

[Paul C. Anagnostopoulos]

T'ai said:

Um, it doesn't matter if someone hasn't passed even a preliminary test. You can do a meta-analysis on the statistics still.

What statistics? The little Russian girl read the newspaper for awhile, then Randi put some duct tape around the mask. Suddenly she couldn't read. No statistics required.

Etc., etc., etc.

~~ Paul

 

[UnrepentantSinner]

This person really needs to work on his trolling skills.

 

[T'ai Chi]

why bother?

Here's why.

Obviously no one has passed any tests. But let's look at all of their standardized scores.

It be interesting to see if

1. the majority of the standardized scores were positive, or

2. the majority of the standardized scores were negative, or

3. there are an about equal mix of positive and negative standardized scores,

to see if there is some overall effect, when averaging across all the tests.

 

[T'ai Chi]

T'ai said:
What statistics? The little Russian girl read the newspaper for awhile, then Randi put some duct tape around the mask. Suddenly she couldn't read. No statistics required.

Etc., etc., etc.

~~ Paul

Um, sure, in cases like that, we cannot get any statistics. In other cases, one can easily get statistics, such as in dowsing underground pipes, film cannisters, anything where someone claims to be able to know the contents of something, remote viewing, etc.

 

[Pyrrho]

As others have said, a lot of these tests don't involve statistics. Generating numbers for the purpose of a meta-analysis would simply be an exercise in assigning values where none existed, and the use of complicated formulae to find significance where none existed.

Given that all of these tests have ended in failure, I don't see any use in the meta-analysis of a series of zeroes.

 

[jj]

Um, it doesn't matter if someone hasn't passed even a preliminary test. You can do a meta-analysis on the statistics still.

There is no reason to assume that the statistics of such apples and oranges mean anything at all.

 

[jj]

This person really needs to work on his trolling skills.

Duno, he seems to be good at getting people to bite.

 

[T'ai Chi]

As others have said, a lot of these tests don't involve statistics.

And as I've said, I realize that there are tests where no statistics are present.

But, obviously, what about the ones where there are statistics?

[/b]
Given that all of these tests have ended in failure, I don't see any use in the meta-analysis of a series of zeroes. [/B]

*sigh*, it wouldn't be an analysis of "a series of zeroes", but rather the standardized scores (of course from the tests where statistics were kept).

The point is to see if most of these standardized scores are positive, negative, or a good mix of both.

If there is nothing going on in terms of significance, we'd expect a good mix of both. Note that if that was the case, that wouldn't necessarily at all imply that something paranormal was going on (as some might want to hint at).

 

[Paul C. Anagnostopoulos]

If you perform a meta-analysis of a series of similar experiments (if there are enough similar ones among the lot), none of which were significant, you'd most likely reach the conclusion that the series was overall insignificant. At least, that's the conclusion I hope you'd reach.

The primary purpose of meta-analyses is to determine whether additional experiments are worth performing. But in the JREF case, additional experiments will be performed regardless. So what's the point?

~~ Paul

 

[billydkid]

Troll??

This person really needs to work on his trolling skills.

I think the term "troll" should be reserved for those who actually deserve. I am not a big fan of Tai Chi's reasoning skills, but I do not remotely think he is a troll. There are genuine trolls in here, but using the word loosely robs it of its significance.

 

[Paul C. Anagnostopoulos]

I agree with Billy. I've been called a troll for being too persistent in pursuing a topic, the exact opposite of what a troll really is.

~~ Paul

 

[T'ai Chi]

If you perform a meta-analysis of a series of similar experiments (if there are enough similar ones among the lot), none of which were significant, you'd most likely reach the conclusion that the series was overall insignificant. At least, that's the conclusion I hope you'd reach.

I agree Paul, and I would expect that too. However, we could talk about how it might turn out all day. It would be interesting to see if our expectations are correct I'd say.

The majority of the standardized scores could be positive or negative for example, and the overall conclusion could be significant.

The primary purpose of meta-analyses is to determine whether additional experiments are worth performing. But in the JREF case, additional experiments will be performed regardless. So what's the point?

I personally wouldn't say that that is the primary purpose of a meta-analysis. According to how I learned it, the primary purpose of a meta-analysis is simply to accumulate experimental and correlational results across independent studies.

And to reply to another post without making another post of my own; I don't pay any attention to the 'troll-sayers'. I find that a lot of the time these people simply cannot make any valid points of their own, so they resort to a label game. I don't even read these peoples' posts after a certain grace period.

 

[Paul C. Anagnostopoulos]

T'ai said:

I personally wouldn't say that that is the primary purpose of a meta-analysis. According to how I learned it, the primary purpose of a meta-analysis is simply to accumulate experimental and correlational results across independent studies.

In other scientific endeavors this might serve some purpose. But with psi, where the experiments are not investigations of underlying theories, the most a meta-analysis can do is encourage you to develop a theory and test it (which is not the mission of the JREF). The meta-analysis does not allow you to reach the conclusion that psi exists, because the experiments were not investigations of a theory of psi.

As I repeat ad nauseam, it is time for a theory of psi. Or give it up.

~~Paul

 

[T'ai Chi]

T'ai said:
In other scientific endeavors this might serve some purpose. But with psi, where the experiments are not investigations of underlying theories, the most a meta-analysis can do is encourage you to develop a theory and test it (which is not the mission of the JREF). The meta-analysis does not allow you to reach the conclusion that psi exists, because the experiments were not investigations of a theory of psi.

As I repeat ad nauseam, it is time for a theory of psi. Or give it up.

~~Paul

So, you wouldn't be interested in the results of a meta-analysis then?

I think evidence determines theories, not the other way around.