Is "peer-reviewed" a distinctive characteristic of high-quality research or not?

Question

"Many matters in academia, such as who gets funding, what gets published and who gets employed are settled by a relatively small group of people." (Lucas: "Legitimacy of peer review" )

Often we hear "peer-reviewed" praised as a feature of high-quality papers in science or mathematics - in particular in modern mathematics where a comparatively small fraction of believers in transfinite set theory have conquered most positions as academic decision makers.

This position is criticized by others, for instance [Henry Flynt: "Is mathematics a scientific discipline?" (1996)]

No matter how much the content of mathematics exploits paradox, mathematicians express dedication to policing their doctrine against inconsistency. Mathematicians do not welcome those who attempt inconsistency proofs of favored theories. [...] I will propose that the main factor in the establishment of "truth" in mathematics is professional procedure and discipline. [...]

Truth is negotiated on the basis of manipulation of import by distorting interpretations. Interpretation takes the form of discarding traditional intentions concerning mathematical structure: the privileged position of Euclidian geometry; the invariance of dimension; the association between integer and magnitude; uniqueness of the natural number series; etc.

From time to time, results are discovered which patently embarrass the conventional wisdom, or controvert popular tenets. [The Gödel theorems.] Then follows a political manipulation, to distort the unwanted result by interpretation so that it is seen to "enhance" the popular tenet rather than to controvert it. [...]

Even if my sense of the situation is right, the appearance of such a professionally compelling proof would be more a matter of packaging and selling than anything else. [...] The biggest hurdle such an attempted proof faces is professional discipline. Whether inconsistency proofs are recognized to have occurred is subject to entirely "political" manipulation.

On the other hand there are also peer-reviewed journals in other fields where the unscientific character is even more suspicious.

http://www.astrology-and-science.com/d-rese1.htm Contains abstracts of 91 studies, most of them empirical, from four astrological research journals. There are 37 abstracts from Correlation: Journal of Research into Astrology 1981-2007 published by the British Astrological Association, 22 from the now defunct Astro-Psychological Problems 1982-1995 published by Francoise Gauquelin in France and (in 1989-1990) by the National Council for Geocosmic Research in the USA, 18 from Astrologie in Onderzoek [Astrology under Scrutiny] 1986-2003 published by Wout Heukelom in the Netherlands, including its precursors 1977-1985 published by NVWOA the Dutch Society for Scientific Research into Astrology, and 14 from Kosmos 1978-1994 published by ISAR, the USA-based International Society for Astrological Research. At the time the first three journals were the world's only peer-review astrological journals devoted to scientific research, whereas Kosmos was more an astrological journal than a scientific research journal, hence the fewer abstracts.

So my question: Is "peer-reviewed" a guarantee of scientific quality or even truth or not, and if not, what ethical standpoint should be assumed and expressed toward it?

Answer 1

I'm not sure what angle you'd like this question answered from, but presuming that, being a site about the study of philosophy, you're asking about the epistemic value of peer review as part of the wider "scientific method", then I think pretty categorically, no, "peer review" is not a guarantee of scientific quality.

Many people outside of academia misunderstand what peer review actually is, conflating it generally with "what other scientists think of the work". Peer review, as far as journal publication is concerned, is a very specific process involving a very small number of "peers" (usually three, only two of whom are independent of the publishing house), all of whom will have been appointed/selected by the editor of the journal, who in turn will have been appointed by the publisher, of which there are basically five (yes five), in the whole world. Thus five CEO's effectively exert their influence on the whole of academic research. More shocking is that the entire system was basically set up by Robert Maxwell as a money making enterprise.

The failure of this system to identify flaws in papers has been well documented by Ioannidis, Sokal, Smith, Bohannon.

Even the editors themselves have at time admitted the failure of the system, as Richard Horton of the Lancet put it

"The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability—not the validity—of a new finding…We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong."

Ex-editor of the NEJM Dr. Marcia Angell

"It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines."

There is evidence of bias in terms of gender, race and institutional prestige have been found (8 out of 9 paper that journals had already published from prestigious institutions were rejected when researcher re-typed them and submitted them again with different institution names.

As a side note (nothing to do with the epistemic value of peer-reviewed knowledge claims), the power of peer-review does not stop at just what papers are read, but in England, it actually effects research funding via the government's RAE assessments (I think there's a new system now, but I've not been involved for quite some time), by which peer reviews of department performance on some very spurious criteria (like "impact") actually determines research grants to those departments.

As Nikolaus Kriegeskorte said in his submission to the Parliamentary review of the peer review system

"A scientific publication system needs to provide two basic services: access and evaluation. Access means we can read anything, evaluation means we don't have to read everything. The traditional publication system restricts the access to papers by requiring payment, and it restricts the evaluation of papers by relying on just 2‐4 pre‐publication peer reviews and by keeping the reviews secret."

He goes on to describe how Open-Access publications and the advances of modern technology can facilitate a more open and robust system which could actually further knowledge rather than restricts it. This was in 2011, to my knowledge absolutely none of the recommendations from any of the witnesses in that review have been adopted to date.