5

Many matters in academia, such as who gets funding, what gets published and who gets employed are settled by a relatively small group of people. This applies quite broadly, from the arts to the sciences.

In the case of funding where public money is being spent on projects, a small group of people are making a decision on behalf of the population at large. The power to make these decisions is granted by the people in a very indirect way, elected representatives choose bodies, which then use a rather complex system of panels to make decisions. There are checks and balances of course.

But is this incredibly indirect representation of the population enough to make these panels legitimate? Are there other factors that legitimacy rests on?

Perhaps institutions responsible for making these decisions and the systems they employ provide a source of legitimacy. Perhaps its only the case when their procedures are out in the open.

Anyway, I'm struggling with this a bit, any suggestions would be welcome.

Lucas
  • 1,791
  • 1
  • 11
  • 26
  • This _might_ be helpful: http://plato.stanford.edu/entries/legitimacy/ and/or http://plato.stanford.edu/entries/scientific-knowledge-social/#SocDirSci –  May 09 '14 at 15:05
  • 1
    OK, I just spotted this, but the first line suggests it is right on the money w.r.t. your question: ["Bias in Peer Review" (PDF)](http://faculty.washington.edu/c3/Lee_et_al_2013.pdf). –  May 09 '14 at 15:13

1 Answers1

2

So there are problems with peer-review, but it's not quite as bad as you are making it out to be here.

Talking about the process of how a submission becomes published will help explain.

First, the author submits an article to a journal for consideration. At this stage, the article is prepared for blind review, so there is no identifying information in the manuscript.

Second, the editor of the journal will decide whether the submission is worth being refereed or not. If not, then the paper is "desk rejected". Many journals get several hundred submissions a year, and there simply aren't enough people to adequately referee them all. So the editor has to make some call about which papers are worth spending the valuable resources necessary to bring the article up to publication quality on.

If editor thinks the article maybe has a shot of getting published, he or she will send the article to referees for comment. Usually the paper goes to between one and three referees.

Different journals can be ranked in terms of who knows whom in this process.

A single blind journal will be one where the referees don't know the authors of the papers they are refereeing.

A double blind journal will be one where neither the referees nor the authors know each others identities.

A triple blind journal will be one in which neither the referees, nor the authors, nor even the editors know the identities of the authors and referees. (These kind of journals require editorial assistants who handle the emails and so forth to preserve the integrity of the process)

Now even triple blind journals are susceptible to failures of the review process. It can just so happen that you get asked to review a friend's paper, or a paper you happen to have heard at a conference. Ideally, a referee in such a position should recuse themselves, but my sense is that that doesn't happen often, just because it's often really difficult to find qualified referees who are willing to volunteer their time, so recusing oneself might be equivalent to getting the paper automatically rejected for lack of qualified referees.

There are problems with actual implementation to be sure, but overall I think the process at triple-blind reviewed journals is sound.

  • Hmm. "[I]n a survey of nearly 1,500 editors in chemistry, a plurality of respondents stated that double-blind was “pointless, because content and references give away identity”" (from the pdf in my comment to the question). Triple-blind isn't (even) mentioned, but that shouldn't make much of a difference. BTW: I don't know what 'a plurality' is supposed to mean here?? –  May 09 '14 at 15:21
  • I guess they probably mean that the largest number, but a number less than 50% agreed with that statement. I'm a bit surprised by that finding, to be honest. –  May 09 '14 at 15:25
  • You're right on plurality. I checked. Also: "This assumption has been tested in a number of empirical studies, which showed that reviewers can successfully identify authors 25%–40% of the time[.]" –  May 09 '14 at 15:30
  • That's really interesting. Although, it does show that the reviewer doesn't know more often than he or she does. Personally, I think the bigger problem with peer review is on the editor's side, rather than the referees. There are too many referees to successfully game them all. There are many fewer editors of important journals, so their personal idiosyncrasies are potentially much more profoundly important. –  May 09 '14 at 15:34
  • I was more trying to ask about what the basis is for giving a small number of people the power to make these decisions, what makes it just. It's especially questionable when the public good is a factor. You might think the process is sound, I don't particularly disagree. It's arguable that an agreement about this in the population is essentially what legitimacy comes down to - then again, this runs into the problem of expertise. – Lucas May 09 '14 at 15:37
  • @shane 1) More likely it shows that some reviewers are good at identification and others are not so good at it. 2) That's why I mentioned that the paper doesn't mention triple-blind. Is it common? –  May 09 '14 at 15:38
  • What makes it just is that hardly anybody is able or willing to do it. To really be an honest editor you have to know a tremendous amount and be willing to sacrifice a huge amount of your own time to sift through the hundreds and hundreds of submissions you receive and make these calls about which things to pass on to referees and which not to. It's difficult, time-consuming, and not particularly well-compensated work. –  May 09 '14 at 15:39
  • @Watson, in Philosophy, I gather that it is becoming more common at the top journals. Two prominent journals that practice triple blind are Nous and Mind. Smaller journals can't afford the editorial assistants, etc. I don't know what the statistics would break down and I wouldn't want to venture a poorly-informed guess. –  May 09 '14 at 15:40
  • I don't see how it being superficially altruistic means that it's fair or, in the case of funding decisions, that they are made in the interest of the population who ultimately grant these powers. – Lucas May 09 '14 at 15:43
  • 1
    @shane I'm thinking of revising this question quite heavily, it will probably make this answer look a bit weird. I have a choice of whether to ask again, or edit. I thought I'd let you have a say in it. – Lucas May 09 '14 at 15:57
  • 1
    Just go on and revise it. If necessary we can revise my response, or hope somebody else comes up with a better one. –  May 09 '14 at 16:32
  • @shane I didn't see this before I posted a revised question. – Lucas May 09 '14 at 17:56
  • @shane http://philosophy.stackexchange.com/questions/11437/can-public-funding-panels-properly-represent-the-interests-of-the-public – Lucas May 09 '14 at 17:56