How to review a journal article
Given that November is Academic Writing Month, I’m reposting this article, which was originally published in 2014 as part of the Centre for Imaginative Ethnography’s Laughing Matters series and is no longer available on their site. Darlene McNaughton, Shaylih Muehlmann, Denielle Elliott, Andrew Ham and Tim Bell all provided input on the post.
Peer review is one of academia’s unifying principles. In a complicated, unregulated and extraordinarily diverse environment, it’s the core value we share. For this reason, the first time they are asked to review a manuscript is a momentous occasion for budding academics. Suddenly, the shoe is on the other foot – or perhaps more accurately, you are now the boot instead of the ball.
In common with the majority of tasks academics are required to undertake,1 how to review papers should ideally be absorbed through osmosis, as opposed to anything as passé as formal guidance or training. But sometimes it’s helpful to have insights from colleagues more experienced in the process. Although I do not claim to have subjected the peer review process to any systematic study,2 I like to think of myself as someone with a certain amount of expertise in being reviewed. Why, just this year I published one paper that received 25 reviews from 16 people at four different journals. How many people can boast that? As an associate editor of an academic journal, and a regular reviewer for a number of others, I’ve also had the opportunity to witness the peer review process from a variety of angles.
It’s important to bear in mind that peer review is not so much a technical process but a disciplining one. However, peer review represents an anomalous kind of discipline, because its subject and object are not distinct. Academics are always simultaneously being disciplined and doing the disciplining. I think ‘Mistress Slavegirl’ captures its essence best: ‘Sometimes I’m the bad teacher; sometimes the naughty school girl. Sometimes I’m the whip, and sometimes the flesh. Giving and receiving pain: it’s what I do’.
Once you have grasped this basic insight, masterful peer reviews will surely follow. However, for your edification I have compiled an overview of four primary review types, distilled from various reviews I have received, observed and, on occasion, written myself over the years. This list is not comprehensive, and many reviews fall outside the types I outline below (I shall have more to say about them later). Instead, I have specifically chosen to focus on those that I believe academics living in the twenty-first century will find most useful.
1) The non-review review
The most useful review in the academic’s repertoire is what I like to call the ‘non-review review’. Generally short, this review is filled with vague and ambiguous statements that don’t really give the author any concrete information on what she or he might do to improve the manuscript. For this type of review, three or four generic sentences should suffice: ‘I am not convinced by this piece’; ‘The tone of the manuscript needs work’; ‘I do not find your argument clear’, and so on. The distinct advantage of the non-review review is that it saves you the bother of having to actually read the paper. Cousin to the non-paper paper,3 it’s an important labour-saving device for any academic wishing to look service-minded and collegial while expending their energy on activities that actually count for something (see #2 for further details).
It’s worth noting that the non-review review may occasionally take a radically different form. This version provides pages of highly specific comments on the paper but entirely avoids any substantive feedback. Although rare, the primary advantage of the long-form version of the non-review review is that it looks like the product of intensive effort, while leaving the editor and author utterly in the dark about your opinion of the paper (thereby allowing you to avoid the bother of having one).
2) The self-promotion review
As we’re all aware, although peer review might be the lynchpin of academia, it has no real ‘CV value’. Reviewing 10 papers – or 50 – will not bring you one whit closer to that promotion. You must focus your attention on activities that generate hard numbers. But if you’re smart about it, you can raise your research profile (and your h-index to boot!) via the reviewing process. I don’t want to sound crass, but as a reviewer you have the author at your mercy, and you’d be a saint (or an idiot) not to take advantage of it. Anyway, everyone knows that this is how the system works: quid pro quo. Demanding that authors cite your work, even if it isn’t relevant to their paper, is your reward for doing the review – a form of delayed gratification for efforts made. Besides, we all know that people deploy citations primarily for rhetorical purposes, so what does it hurt for them to cite your work for instrumental purposes instead?
Given that this a broadly recognised function of the peer review process, I can hear you thinking: ‘Won’t that alert authors that I wrote the review?’ Especially if you’ve dished out some ‘tough love’ (see #3), it would be impolitic to inadvertently reveal your identity, given that most academics have long memories and hold grudges. The secret is to disguise your identity by embedding your name in a list of references. Of course, authors will very likely suspect that one of the names on the list is a reviewer of the manuscript, but they won’t know for sure who it is.
3) The tough love review
The life of an academic, as we’re all aware, is about critique. As Bledsoe and colleagues note, ‘Our business is to read research proposals, journal articles, student papers, and to find fault’. No one will survive the illustrious halls of the academy for long if they are incapacitated by critique, so you are doing the author a favour by giving them some tough love. The point is that you can’t make an omelette without breaking a few eggs. If you can’t handle the heat, get out of the kitchen. It’s the grit that makes the pearl. Whatever doesn’t kill you makes you stronger. No pain, no gain. Spare the rod and spoil the child. The naysayers might have their ‘data’ and ‘studies’ and so forth proclaiming that tough love is ineffective and counter-productive, but who are we to argue with dozens of folksy aphorisms?
As someone who has been the recipient of a fair amount of tough love over the years, here are a couple of phrases I think you’ll find helpful. ‘This looks like a graduate essay cobbled together at the end of semester’. This one holds a special place in my heart because it was the opening line of the first review I ever received.4 I mean, I didn’t submit anything for more than two years afterwards, but look how much it helped in the long run! ‘The tone of the article reveals one of the most dramatic lack of even-handedness in its presentation of arguments that I have read in my entire career’. This was the clincher of a recent review I received and is a variant of ‘This is the worst article I’ve ever read’. Although such phrases might appear hyperbolic, and I would certainly caution the reader not to overuse them, sometimes the big guns are called for. This might be because you utterly disagree with everything the author is saying and feel it is your moral duty to ensure the paper never sees the light of day. Or it might be because you’re having a rough week.5
With enough skill and practice, it’s possible to compact the tough love review into a few pointed sentences, such as the following: ‘This paper must be rejected. It is contentious, mischievous nonsense. The argument is outrageous, and badly presented. This is a spurious, frivolous rant. I am astonished that you considered it worthy of sending for review. You have wasted my time and yours’.6 Indeed, true masters are able to distill their sentiments into a single-sentence review: e.g., ‘The arguments presented in this paper are untenable’ or ‘This is utter rubbish’. While some of you may worry that such reviews seem unduly harsh, it’s your academic duty to provide such feedback: you got (get) tough love; now it’s time to return the favour.
4) The Persian Flaw review
Let’s say you’re reviewing a paper and – Heaven forbid! – you really like it. Bearing in mind your professional responsibility to find fault (see #3), I would suggest you utilize the Persian Flaw review. This type of feedback is generally positive, but involves the reviewer taking exception to some passing point or minor aspect of the paper. It might be that they dislike your characterisation of someone’s work (possibly theirs); it might be that they want you to elaborate on a tangential point. The trick is to focus on some minor aspect of the paper, and then bang on about it throughout the review process, elevating its status to a major amendment.
For my own part, I have an intense aversion to acronyms, so I make a point of berating authors for this in reviews.7 This might seem like a minor offense, but all manner of academic sins can be symbolized in the use of acronyms: laziness (what, you can’t even be bothered to write out the word in full?), false reification (should we really be talking about the global war on terror, or ‘GWT’, as a thing?), and so on. Now, this might seem rather petty, but recall that the purpose of this type of review is primarily philosophical. Just like the flaw incorporated into the design of Persian rugs, it serves to remind authors that perfection is unattainable and that no matter how well they’ve done, they can always do better.
It’s worth bearing in mind that these four reviews are ideal types. In reality, it’s common to see several combined in a single review (e.g., the non-review review + the self promotion review). But whichever form you choose, try to avoid sliding into a substantive review (a.k.a. the ‘review review’). I have seen these more often than one might expect given the conditions under which we labour, and have slidden into them myself on many occasions – although I assiduously guard against such tendencies. What purpose does it serve to have as your goal nothing more than intellectual engagement with someone else’s arguments and ideas? This kind of naïveté has no place in contemporary academic life, so is best nipped in the bud.
Teaching, marking, presenting papers, writing them, and so forth.
Unlike the venerable Cochrane Collaboration, which was recently impelled to review peer review. Resolutely undeterred by the fact that this required them to engage in a process that looked suspiciously like the one they were critiquing, they have once again proved the sorts of extraordinary insights to be garnered through the unquestioning transposition of the principles of evidence-based medicine into all manner of social phenomena.
The ‘non-paper paper’ is the paper that essentially duplicates one you have previously published – with slight modifications to its form and content. Any skilled academic should be able to get at least three publications out of every paper they write.
Most of us can still quote chapter and verse from the first review we ever received, especially if we got some tough love – as Rosalind Gill illustrates.
As millions of internet users can attest, life offers few opportunities for anonymous venting, so you should take them where you can.
Thanks to Simone Dennis for allowing us all to benefit from a tough love review she recently received from a clear master of the form.
So if you’re reading this and you have received a review where you were told to tone down the acronyms, then, yeah, that was probably me.