I decided to read a grant application on the plane from Rome. As I was reading it, I got more and more frustrated with what was written. Basically, the project was all over the place. From strange and unaccounted for concepts, through bizarre associations between groups, all the way to a reality which I simply didn’t recognise. The review was not very difficult write, well, perhaps apart from the fact that it was very negative. That’s life, though.

The review got me thinking about reviewing. As we teach our students (and staff) how to write books, articles, grants, I have not heard of any advice how to write a journal review. Well, any formal advice. There is, of course, the excellent Writing for research blog with a recent post on how to comment to which I will come back in a moment.

So, I was thinking, I couldn’t remember the first time I had been asked to review an article for an internationally refereed journal. I do remember, however, the first few times, so to say. To be completely honest, I was first and foremost frightened. I was still very junior, and young. I had only just had my first publishing successes, and someone (with a ‘big’ name) asked me to comment on others’ work. Who am I to do that, I thought? I know nothing, or almost nothing. Really, I was mostly scared, I would make a fool of myself. And that was my first experience of reviewing a paper for a major journal. An experience, I later learnt, which was not uncommon, though not many senior people admit to it.

As the experience of reviewing means that you mostly write negative comments, my first decision was whether to actually write them. After all, I was thinking, negativity is not exactly very nice, so perhaps I could sugar the review a little bit…This was one of my first decisions I took as a reviewer – I decided that I might make a fool of myself, but at least I would be an honest fool, without needing to second-guess myself. That decision, taken well over 20 years ago, has been one of very few completely unbroken promises to myself. I have never written a dishonest reference. I always have and still do write what I think. Incidentally, I have had some trouble because of this.

So, how to write a review? Patrick Dunleavy rightly says that commenting on others’ work is very difficult. Recently, he wrote some advice on how to comment. He reminds reviewers not to make over-strong or un-evidenced comments. And of particularly note is advice

• to list any points of agreement,
• mention anything you learned
• after which you can offer a rebuttal.

Well, I mostly disagree with this advice, but only in the case of journal reviewing. For commenting on a PhD draft, I agree. So, I disagree, because I really don’t care whether the reviewer agrees with me and I assume that authors don’t care if I agree with them. I care a bit more whether the reviewer learnt something, but only marginally more. I agree, however, the reviewers should reflect on what they themselves say.

So here is my advice, based on over 20 years of reviewing. Let me start with what I do. Is it sensible? I don’t know. However, over wine, I was told by an editor of a major journal that I am coldly fair.

1. I always start with a brief assessment of the article as a whole, particularly its topic. Is it worth reading? Here I often say the one (necessary) good thing….
2. Then I go to the methodology, I go through it with a fine-tooth comb. That’s the backbone of any study – if it makes no sense, there is no point in the study. It has happened to me on a number of occasions that I stopped reading the article after the methodology section. There was no point in going further.
3. Only then do I start considering the research context, the literature.
4. Then I go to the main body – as I mostly am asked to review qualitative stuff, I look at the analyses. There are, incidentally, way too many of those which are based on not much more than the author’s whim.
5. Then it’s conclusions. Does the author understand what their study allows them to say? Alas, more often than should be the case, they don’t.
6. I am also looking for how the author accounts for what they did, an understanding, the process.

Here are some things I avoid.

1. In contrast to the occasional review I get, I don’t tell the author how I would have designed/done/written the article. No one cares and my perspective is hardly key. And if I am so keen, I can do the research myself.
2. Only in extreme cases do I suggest that my work should be taken into account. I find it embarrassing, even though the review is anonymous. I also do remember that the fact that I would quote someone, means not much more than that I would quote it.
3. I avoid forced positivity. When I started working as an academic after graduating, I worked with a professor who decided to help me with my writing. So whatever I wrote, he commented on. And he was ruthless. As he was (and still is) extremely witty, he made me laugh at myself. It was almost cruelty. But, gosh, did that teach me how to write! I really hate when people try to ‘spare my feelings’ – the pain of total rejection is sharp, but short. Much better than living in the delusion of grandeur.
4. Often editors offer the possibility of addressing them directly. I never do it (well, I did it once or twice – the papers were so bad I had to stress it) – I think the process should be transparent.
5. I do not seek to agree with the author, I (try to) assess their work on the basis of the assumptions they make.
6. I never comment on something I don’t know about, or even something I am not certain about. If I am not familiar with the current literature (I still get papers on deception for review – even though I am no longer in the field), I don’t comment on it. There is always plenty to comment on in methodology.

Finally, I do remember that I am to offer an honest peer assessment. I try hard to make my reviews helpful, but it is not my job to make it pleasant. Trying to point out the weaknesses, I think I help the author to write a better article. And this is how I see my role. Unfortunately, more often than not it means taking the article apart. When I do it, I always bear in mind that honest (and courteous) critique is good and desirable. Critique that pulls no punches is even better.