Senior scientists "usually wrong"

[Hellbound]

It makes no more sense to reject criticism of peer review than it does to keep using an uncalibrated test instrument.

And it makes no sense to b**ch about your uncalibrated instrument unless there is a way to better calibrate it.

The peer review process is not perfect, I think EVERYONE here has stated that and agrees with it. The point is that criticising it without any forward progress is not science, either...it's flat out two year old whining.

The phrase constructive criticism came about for a reason...because most criticism isn't. Pointing out the problems is all fine and dandy, but pointless unless one can find a better way to do it. You basically stated that whether there's a better way is irrelevant, we should criticise anyway. To what purpose? We understand that it is not a perfect process, but it works over time in most situations. There are more bad papers that get through than good ones that don't, but that is why we have additional things such as repeatability and replication. A published paper, no matter who published it where, is usually not given much weight until the experiemnt is sucessfully reproduced. Bad data, omitted data, and incorrect results do get weeded out.

Trying to position peer review as the only safeguard in science is like trying to say calibration is the only step to measurement. You're missing the bigger picture (not to mention my earlier point that criticism without purpose is just that...purposeless).

 

[pgwenthold]

The number of peer reviewed articles that are showing up with 'manipulated' research, or orginal raw data that gets 'lost' is unacceptable. These occurences should be as close to zero as possible, not just shrugged away.

Argument by assertion. Can you provide any reason to think that it is _not_ "as close to zero as possible"?

The key word here is "possible."

Or, as others have been saying, you got anything better in mind?

If, as you claim, the peer review system is not making the bad occurences as close to zero as possible, you must thing it is possible to do better. So how can that be done?

As I noted before, people are constantly trying to tweak the system to make it better, and in the end, nothing they do is really making a difference, suggesting that big improvements (if any at all) are not all that easy to come by.

Personally, I don't even blame the system for the mistakes. I blame it moreso on lack of diligence by the reviewers. Basically, it takes a lot of time, and there is a lot to do. Considering that every paper needs something like 3 reviews, that means that on average you will see 3 times as many papers to review than you write. That is a massive time burden (a review generally takes me the time to read the paper, digest it for a few days (ok, not much of an issue), and then about a day to write), and it is easy to not take the job seriously as it should be taken.

I see it in the same way I see teaching: the main problem is not the format (lectures, group study, etc), but people who don't do it well. You can play all the games with teaching rubricks that you want, but the most important thing to do is to have good teachers.

 

[jj]

Well, I am a "senior scientist" and I am a classic counterexample to the utter lies so unethically stated in the original article.

To reply to the (lack of) substance, the gentleman's claims read very much like those I've heard from any number of people who submitted something without doing their homework. It reminds me of more than a few people here, frankly.

I've submitted papers that very seriously challenged the standard paradigm and had them accepted. I did, however, show solid results, evidence, and explain the mechanisms that led me to do what I did.

I will note that perceptual audio coding (MP3, AAC, WMA, AC3, DTS, etc) was once considered a "fringe field", and that people who attempted to work with what we knew about human perception (in coding) were regarded as "more than a bit off" (and to be honest, most of them were).

None the less, solid work and solid evidence that one COULD use human perception won academic, business, and publication acceptance. If you challenge the "conventional knowledge" have some evidence and a demo, basically.

Such papers still remain somewhat of a pain, because they split across 4 different domains of knowledge (psychoacoustics, signal processing, noiseless coding, and computer science), and most reviewers, until recently, were boggled by one or more of their non-specialities. (Now we have a core of 4 or 5 reviewers who can cope, thankfully...)

Now, if I could only get the same thing to happen in multichannel audio production .... (Hint, any front-based multichannel signal should have more energy in CENTER than anywhere else, and Fletcher and Snow showed this in about 1930 or so)

 

[rockoon]

Huh? Who's stopping you? Is this the same senior scientist from the NSF that keeps stealing my car keys?

So your contention is that you can force a publication to publish your work?

 

[Hellbound]

So your contention is that you can force a publication to publish your work?

And here, ladies and gentleman, we have an example of both "Shifting the Burden" and "Strawman" fallacies...a two for one deal, if you will.

Now, if you'll please move on you can see the next exhibit shortly...

 

[crimresearch]

"Argument by assertion. Can you provide any reason to think that it is _not_ "as close to zero as possible"?
The key word here is "possible."
Or, as others have been saying, you got anything better in mind?"

"One or more statistical errors appeared in 38% of Nature papers and 25% of BMJ papers...up to 4% of statistically "significant" results reported in these papers may not be."


Do you not even bother to read before you post? Or do you just figure that all you have to do is lie, and reality will magiaclly shift to conform to your notions?

If you think that these numbers, and comparable ones from similar audits of other scientific disciplines and peer reviewed journals are 'as close to zero as possible', you've failed the basic math test for admission to adult discourse.
Bye bye and thanks for showing your inability to comprehend simple word and number statements.

--------------------------------------------------------------------------

" The peer review process is not perfect, I think EVERYONE here has stated that and agrees with it. The point is that criticising it without any forward progress is not science, either...it's flat out two year old whining."

'Everyone' would even include the post where I said it isn't a perfect system earlier in the thread. And pointing out specific areas which could be improved as I ( and others) have done isn't whining, it's called..let's see, how about 'construcitve criticism'?

"A published paper, no matter who published it where, is usually not given much weight until the experiemnt is sucessfully reproduced."

Realy?? I would love to see you back up that reckless asser6tion with some evidence, given the number of published articles in which replication gives different results, AND the aforementioned problem of researchers failing to share the original raw data, which has been explicated in a variety of studies.

I would suggest that those who have been pontificating without even bothering to read what has been said, take an advanced statistical course which involves assessing the sort of research currently being published in just about any discipline.
It should prove a real eye opener.
And it will give you several ideas for what could be done to improve the process.

Or just stay there with your heads in the sand...your loss.

 

[pgwenthold]

"
If you think that these numbers, and comparable ones from similar audits of other scientific disciplines and peer reviewed journals are 'as close to zero as possible', you've failed the basic math test for admission to adult discourse.

OK, so tell me, how close to zero is possible?

Especially given the second requirement that you cry about not having enough opportunity for earth-shattering breakthroughs.

Moreover, of the errors that have been identified, how many of them undercut the thesis of the articles in question? "Up to 4% of the significance claims may be unjustified." Considering that things claimed to be significant to the 95% level are wrong, what, 5% of the time anyway, such a level isn't a disaster by any means. Besides, how many of the articles in Nature are statistical studies in the first place?

I note you have moved the goal posts once again. It went from "manipulated" data and "lost" raw data to "statistical errors."

Moreover, what process do you propose to prevent the statisical errors that you lament? I have a suggestion, but you won't like it - send the paper to an independent reviewer who knows a lot about statistics and can evaluate the statistical claims. Whoops, that's peer review...

 

[Hellbound]

'Everyone' would even include the post where I said it isn't a perfect system earlier in the thread. And pointing out specific areas which could be improved as I ( and others) have done isn't whining, it's called..let's see, how about 'construcitve criticism'?

Um, that's just it. You haven't pointed on any specific areas that could be improved, or given any ideas as to how they could be improved. As other have pointed out, the system gets refined continually, and changes are being made. Complaints without offering any reasonable alternative accomplish nothing, it's already well-known that the process is not perfect. Your criticism is not constructive.

Realy?? I would love to see you back up that reckless asser6tion with some evidence, given the number of published articles in which replication gives different results, AND the aforementioned problem of researchers failing to share the original raw data, which has been explicated in a variety of studies.

Okay. You're really p*ssing me off here because you're either a d*ckhead or just too dense to breath without a note to remind you. Show me a SINGLE case of one published research paper changing accepted scientific thought. Just one. A single case. A paper getting published is the first step towards becoming a part of mainstream science, not and end-all stamp of approval. Your examples showcase EXACTLY the point I was making. There are articles published with different results (showing a lack of bias), and research were raw data isn't shared is regarded lower than more upfront research. Only when multiple published papers, all with good data and good procedure, are published does the idea start to take hold in mainstream science. The review process is an introductory screening, not the end-all and be-all of scientific acceptance.

I would suggest that those who have been pontificating without even bothering to read what has been said, take an advanced statistical course which involves assessing the sort of research currently being published in just about any discipline.
It should prove a real eye opener.
And it will give you several ideas for what could be done to improve the process.

Or just stay there with your heads in the sand...your loss. [/B]

No one has their heads in the sand. As I stated, EVERYONE here admits, rather readily, that the system of peer review is not perfect. Everyone admits that if it could be improved, that would be just fine and dandy. However, what YOU fail to understand is that A) peer review is only one part of science, and IMO not the major component B) The processes of peer review are constantly being evaluated and modified to improve it C) The vast majority of scientists already undertsand the limitations and failings of the peer review process, and take that into account when studying peer-reviewed journals D) A single incorrect result published in peer review is highly unlikely to make a change in science; any major issue would require a systematic publishing of multiple papers, all on the same specific effect/phenomena, all with data that is bad in a similar way, and all reproducing each others results. Such a situation, while not impossible, is highly unlikely. and finally E) You've yet to offer a single, constructive or useful suggestion as to how to improve the process, despite your claim. You are ina fit of whining. If you had come here criticising the problems and suggesting alternatives, there might be something to discuss. So far, you've posted nothig new or meanignful.

I suggest that those who have nothing constructive to add and simply want to b*tch should first understand excactly wtf they are discussing, and the process about which new discoveries get accepted into science. I would also suggest that someone take their own advice, and let us know the "several ideas for what could be done to improve the process" so we actually have something to discuss.

You aren't being ridiculed for criticising peer review...you're being ridiculed for bringing nothing of value to the discussion and acting like a general prat.

 

[jj]

...
I suggest that those who have nothing constructive to add and simply want to b*tch should first understand excactly wtf they are discussing, and the process about which new discoveries get accepted into science. I would also suggest that someone take their own advice, and let us know the "several ideas for what could be done to improve the process" so we actually have something to discuss.
...

Hear, hear!

The facts that the original thread-starting article by that twerp says that new ideas are rejected, and then the supporters of the premise point out that contradictory and inconsistant articles get published are obviously in opposition and the supporters of the premise, by citing the fact that contrary results appear, have entirely, completely shot themselves in their own collective feet by pointing out that work contradicting the "general knowledge" DOES get published if and when it's published in a way that can be tested and examined. That contradictory work is the "new stuff" that the original twerp can't seem to realize exists.

Huntsman, dunno who you are, but thanks for taking the point here.

Oh, to be clear, the "twerp" is the person who wrote the article that Iain quoted.

 

[jj]

Really I don't understand the criticisms of Calder at all when talking about the peer review of papers. I don't have any figures handy, but I would guess that most papers that get rejected during peer review are those that are either obviously wrong, or aren't significant advances on research already done.

I will have to say that there is a third catagory, papers for which the reviewers didn't understand and didn't dig into, that were in fact not "wrong" merely different.

On the other hand the system seems to correct for those rather nicely, if my own experiences are any example.

My most cited paper was first rejected in exactly the form it was finally published in.

 

[rockoon]

And here, ladies and gentleman, we have an example of both "Shifting the Burden" and "Strawman" fallacies...a two for one deal, if you will.

Now, if you'll please move on you can see the next exhibit shortly...

Is a completely accurate study that conforms to all scientific standards which exposes some new understanding in a given field to which a peer reviewed periodical concentrates on certain to be published?

There is no chance at all that there is bias?

Hmmm?

So much for critical thinking. Throwing around claims of logical fallacies only makes you look smart for about 5 seconds. Then we look back at the correspondence that is happening and your comment sticks out as a self serving spout of incoherence. Had a bad day? Did it make you feel good? I'm glad.

Perhaps you should read the thread next time?

If you think any piece of published research is flawed you are of course free to prove it.

But you are of course not necessarily free to publish that proof in a peer reviewed publication.

Huh? Who's stopping you? Is this the same senior scientist from the NSF that keeps stealing my car keys?

So your contention is that you can force a publication to publish your work?

And then you:

And here, ladies and gentleman, we have an example of both "Shifting the Burden" and "Strawman" fallacies...a two for one deal, if you will.

Day just got worse?

 

[rockoon]

In 1997, THIS article was published about this subject:

... It is argued here that peer review as now undertaken by most scientific journals stifles scientific communication, slows the advancement of knowledge and encourages dishonest behavior among referees. ...

And I seem to remember a case a few years ago - the NSF investigated a researcher who was also active in peer reviewing - A proposal for a research grant being reviewed by this guy was turned down based on his opinion. This guy then submitted his own proposal for funding which in many places quoted verbatim that proposal he made sure got quashed. He was then found to have stolen other peoples work previously as well.

I'll see if I cant dig up a link to that story.

 

[crimresearch]

"You've yet to offer a single, constructive or useful suggestion as to how to improve the process, despite your claim. You are ina fit of whining. If you had come here criticising the problems and suggesting alternatives, there might be something to discuss. So far, you've posted nothig new or meanignful..."

You might want to see someone about those hallucinations.
Note that rational people are having a discussion about the very things that you deny are being discussed, and no one is interested in your 'outrage' over the things you imagine are being posted.
It is the difference between discourse and the rantings of an obtuse crank.

 

[Hellbound]

You might want to see someone about those hallucinations.
Note that rational people are having a discussion about the very things that you deny are being discussed, and no one is interested in your 'outrage' over the things you imagine are being posted.
It is the difference between discourse and the rantings of an obtuse crank.

Huntsman looks around in befuddlement

Ah, I see, you must have forgotten your note.

Please, point to any single instance in this thread where you've posted something constructive with regards to reforming peer review. I just re-read it, and I can't seem to find that particular post. No matter. You missed the point again. Ah well, such is to be expected.

The difference between discourse and rantings is whether you actually listen to and are capable of comprehending others points of view, and then discuss those. So far, you seem to have failed, specifically at understanding. Thank you for playing!

rockoon:

I find it HIGHLY unlikely that every journal within a given field would refuse a persons article. The point others were trying to make is that if an article has merit, it can (almost always) get published. If one journal turns it down, others can publish it. NO ONE was suggesting that you can force a journal to publish your work (there was your strawman). A completely accurate study that conforms to all scientific standards which exposes some new understanding in a given field to which a peer reviewed periodical concentrates on is highly likely to be published, and no one(typically, at least, unethical people exist everywhere) actively prevents publishing. The issue here is that some are trying to take a matter of chance and paint it as a systematic denial, something that is false and misrepresentative of the process.

I've actually had a pretty good day so far, by the way. And yes, I am feeling rather well about myself I have read the thread...you, apparantly, have read both the thread and the minds of the posters. Thank you for playing also

 

[jj]

Is a completely accurate study that conforms to all scientific standards which exposes some new understanding in a given field to which a peer reviewed periodical concentrates on certain to be published?


Pretty much, indeed it is. Journals do compete, and an editor would be VERY displeased with reviewers who show more than a normal human's rate of unnecessary displeasure.

I certainly have overruled reviewers, editors, authors, etc. Curiously enough, almost always in favor of publishing, too.

I'm hardly alone. If something looks like it might have some sense to it, then it ought to see the light of day.



There is no chance at all that there is bias?


Surely you realize that every journal I have ever used has multiple reviewers, yes? I have had reviewers come in very, um opposed. Then you get another opinion or two.


So much for critical thinking. Throwing around claims of logical fallacies only makes you look smart for about 5 seconds.


You, on the other hand, are starting to show an agenda here, Rockoon. Just like your deliberate picking of an argument in the star thread a while ago, here we see you passing off insults for argument.

No system is perfect. Have you any suggestions as to how to improve this one?

...

Huntsman, this dude has a record of hit postings. PM me if you want to know more.