We place great faith in peer review as a control over what is published and what is not. I have been extensively involved in the whole peer-review experience as an author, as a reviewer and more latterly as an editor. Being an editor opens a window on the process that very few get to see. The way in which reviewers are selected is perhaps something to be shared and analysed, as there are some important points for all editors, associate editors, regional editors, local editors, etc to consider.
Some scenarios spring to mind:
Professor Ed Itornchief has been emailed by the online manuscript processing system to alert him to some new paper submissions. Poor Ed, his journal covers a very diverse range of subject material. Some of it is very familiar to him, but other elements are way outside his comfort zone. Let’s peer into the Professor’s mighty mind to follow his thought processes.
“So, shall I make another coffee, do some exam marking, or look at these journal submissions. Suppose the journal better come first today. Ah, so what have we got here? Four papers submitted, that’s good news as it has been a bit quiet for the journal over the last week or two. What’s the first one?
Effect of vermistat on body weight in young adult men. Ah good, I know something about this. Who would be a good reviewer? Well there is John Smith in my own university. This is in John’s area and I know he will turn it around quickly for me. Who else? Nina Negative at KFC or Terry Tardy at Queens. Terry probably knows more about this, but he is so slow. Let’s go with Nina, though I expect she will probably recommend rejection. She always does!“
OK, so the Prof was happy with this paper and made referee selections without too much trouble. Clearly he wasn’t fair and equitable though and made decisions based upon speedy turnaround of the paper, rather than giving the authors a decent chance of publication. Still, at least he selected people with established expertise in the area. Back to the Prof and his thoughts.
“What’s next then. Ugh, qualitative research. I just don’t understand this at all and don’t know who to approach. Time to do a search using the journal database. Keywords; qualitative. children, meat. What do I get? Oh, not very many here, just three in fact. Still, at least there was something there, I hate it when it comes up with nothing. Who to choose, Paul Brown, Mike Jones or Hermione Granger? Ha! Hermione Granger! That’s funny- got to choose her! I bet she is a wizard reviewer.“
Oh dear Prof! He has gone for a classic editorial bias here and omitted to make some basic checks. All three potential reviewers on his database came up on the search because he entered particular search terms. The Prof forgot to make any checks on whether they are really experts in this field. They could be anybody. In the end he made one selection just because the name was entertaining. Two papers to go. Can the Prof do a better job with the next one?
“Getting bored now. I wonder if there is any cake going in the coffee room. No, come on Ed, be focused. Hmmm…. This looks like a tricky one to find reviewers for. Adipose tissue DHA reserves in adolescent males of the Cheyenne. I could just reject it and save myself the bother. No, that wouldn’t be fair, this must be of interest to our readers. Try the data base I think. Adipose tissue; DHA; First Nations. Oh no! No hits. How about First Nations? Still nothing. Stupid database. This is a pain, but I can get round it by doing a PubMed search to see if there is anything similar out there. Bear with. Bear with… Yes! There are a good number of similar works. But… most of them are by the authors of the submitted manuscript… There are some others though. I will invite those authors to review as they must be experts in this field. Sorted!“
I quite like what the Prof is doing here. He has gone to some effort to find appropriate reviewers based on previous publications. That is a good approach. What he doesn’t have time to do though is establish whether the papers published by his authors 1) are any good, or 2) whether there may be a chance that these related groups of researchers are collaborators (conflict of interest) or bitter rivals. One last paper for Prof. Ed to look at.
“So, what is this last one about? A community based weight loss initiative. This should be easy to find reviewers for. We get so many of these submissions. Ask the database… Weight loss; intervention study. Yes, as I thought, I have 87 possible referees here, across 9 pages of search. Just choose the first two, Arial Aaronson, Abby Abrahams. There! Job done and back to my paid work.“
Dear oh dear. The Prof has fallen straight down the other obvious hole. When searching for reviewers on a journal database it is really important to make informed choices. Find out more about the reviewers listed. Who are they? What is their track-record? Are they really experts? Could they be closely linked to the authors? Moreover, don’t just choose the first two on the list- go through the whole list and choose the most appropriate. I find myself wondering how often researchers whose surname begins with the letters A-D get invited to review, relative to those whose begins with the letters V-Z (funny names excepted, of course).
These are every day possibilities for all journals. All have the capacity to undermine the utility and integrity of the peer-review process. How many editors take the time and care to avoid these potential pitfalls? How many don’t? And how does this impact on the quality of scientific publishing?