A discussion on professional journal refereeing has begun on Leiter Reports (here). The discussion was spawned by an email to Brian Leiter from an anonymous young philosopher, and Leiter posts a snippet of the email in the post.
The young philosopher’s primary concern is how does one review a paper without being too harsh or too lenient? I encourage you to check out the discussion.
These are my random thoughts on the matter:
A commentator (J. Bogart) raises an interesting issue: should the proposed article be judged on the strength of the argument or the quality of the argument? I gather that some wouldn’t see much difference between the strength and the quality of an argument. By strength, I just mean that the reviewer agrees with what the article’s author concludes, i.e., the view the reviewer thinks is the correct view. Philosophy advances through debate. Alternative positions ought to be revealed to promote debate. So it hardly seems right for a reviewer to judge that an article be published because it’s the view with which s/he agrees.
I’ve had the misfortune of receiving reviewer comments composed of his/her opinion and not composed of some constructive criticism. For example, “I feel that the author hasn’t read x, y, and z, though these works have been cited in the references. Sure, the author makes a good argument. But I disagree with it.” Hmmm… isn’t this a bit juvenile? Why do I care whether a reviewer disagrees with my conclusion? I didn’t get into philosophy to agree with anyone/everyone.
Don’t get me wrong, opinions of agreement and disagreement are important. Reviewers should be cognizant of the publication’s aims and goals. If the article doesn’t fit the aims or goals of the journal, then it shouldn’t be published in it. Reviewers — if they don’t already know what the publication’s aims and goals are — should familiarize themselves with them. Check the journal’s website, read a few articles that have appeared in the journal, and scan the table of contents for the journal over the last ten years (or so) if the reviewer’s not already familiar with the content. Reviewers should alert the author if the article’s content is not conducive with the mission of the journal. And — more importantly — the reviewer should recommend other journals more fitting for the article.
An article containing a “quality” argument is one that has fewer gaps than what you might find in a conference paper or dissertation. Similarly, the author should attempt to address any gaps s/he believes others would find in his/her argument. To not address worries anyone might have with the argument would be detrimental for the paper.
A good review evaluates the merits of the argument in the paper. The review should ignore his/her own opinion on the subject when deciding whether the article is worthy of publication.
There was a similar discussion in English a while back—mostly complaining about the tendency for reviewers to re-write articles they’re reviewing, which in turn slowed down the review process and generally gumming up the works. I recently got some comments from a journal that said the following:
1) I can’t believe you relied on X. X has demonstrable errors of fact. (I only referred to X in a footnote, and only by way of saying “X has discussed this.”)
2) OMG IT’S TOO LONG! Cut it in half and it’s publishable.
3) We just published something like this 10 years ago. (My essay was an extension of that earlier study)
Wow! I figured that there must be some cross-disciplinary issues.
In the case of some reviewer re-writing the article they’re reviewing, do you think that this is an egregious form of plagiarism? Of course, if articles are blind-reviewed, how would you know that the reviewer re-wrote the paper? Is it apparent?
(1) and (3) happen too often. I’ve heard others complain that the reviewer latched on to one or two minor points while ignoring the major thesis of the paper. Absolutely ridiculous!!!
Oh, what I meant was that they are unable to comment on the essay without trying to transform it into The-Article-I-Would-Have-Written.