The review I wanted to write…
What happens when people who know better still try to publish their garbage?
I got excited when I recognized the paper’s highly-esteemed authors. One of them was even the dean of a top school of public health. I had reviewed so many bad papers this year, I really needed a good one, just to restore my faith in this whole thing.
Then I started reading. Disappointing doesn’t begin to describe it. I’ve written plenty of reviews where I disagreed with how the authors dealt with a problem — and lost more than a few of these arguments in the process (to my benefit). But this wasn’t that.
By the time I was done, the review was almost three pages long, much of it pointing out how they ignored fundamental problems such as measurement error, selection bias, confounding, and missing data. Then there was their misuse of p-values—you know, run loads of models and then highlight the few estimates where was < 0.05 and build a (new) story around them. And even if they had used the best possible analytical methods, the data were still too limited to provide useful insights, especially when coupled with their superficial approach to theory and context. There were even lots of typos. It was a bad paper.
It ended with a funding statement indicating that the first author was a PhD student. With that in mind, I was especially careful to make sure my assessment was constructive and kindly worded, with lots of references to help them better understand and address the problems in their paper.
But there were other things I wanted to say to the more accomplished co-authors from the prestigious institutions. Things that weren’t as kind. Things that couldn’t — and shouldn’t (?) — make into into the review. Things like…
What the f**k were you thinking when you submitted this garbage!?
I’ve seen so many papers like this. Completely unsound, with an early-career first-author from a prestigious institution followed by a list of highly-esteemed co-authors. Again, I’m not talking about a paper with errors or flaws. Everyone one of us writes those kinds of papers. I mean a total train-wreck that leaves you scratching your head. You’ve probably seen them too, but we keep our mouths shut, despite how harmful this behavior is. We wouldn’t want to impugn the sanctity of peer -review! So if you’ll bear with me, I’d like to get this off my chest, at least once.
So how does this happen and whom does it hurt? How do world class experts manage to submit such a terrible paper? The most cynical explanation is that the co-authors care so little about the content of said paper that they can’t be bothered to read it closely, even one time. The alternative is that some of the top epidemiologists in the world don’t know how epidemiology works.
So here I am, suggesting that senior people and research leaders are just rubber stamping scientific papers. It’s an outrageous idea. But are you shocked? I doubt it.
No, we’ve all accepted that the goal is to publish anything and everything and see what sticks. More students. More “collaborations”. So many papers that no human can possibly be paying very much attention to any of it. And we all know where the incentives lie.
And maybe I can have some sympathy for people earlier on in their career, like the first author, perhaps “playing the game” they’ve been taught, trying to keep a job (though I’m not too sympathetic). But some of these authors have hundreds of papers and thousands of citations. One has an H-index approaching triple digits. They hold high-positions at top schools. If this is a game, they’ve already won — and If we can’t expect people with permanent jobs and six-figure salaries to do the right thing…I despair.
I think there are other, related factors at play of course, but none of them are admirable. I suspect that with so many authors, everyone assumes someone else is paying attention. I’ve probably been a bit guilty of this one myself (though not like this). Could I have read that paper more closely? Could I have checked it one more time?
And some people wrongly view peer-review as some kind of editing service. You just submit your paper and get a free review. Simple. This probably works especially well for senior academics from fancy places who are less likely to get desk-rejected. And why not? There is literally zero accountability for submitting trash, because everything is done in secret.
So let’s think about who suffers when this happens.
The first victim is yours truly, who gave up several hours to read and write a coherent, constructive review of the paper. Clearly any of these super-star co-authors could and should have done the same before submitting the paper. So well-played for getting me to do your job for you under the guise of service to the scholarly community. Guess what? That feels like a giant F**K YOU DARREN. As if I don’t have other projects and responsibilities. Like I wouldn’t like to spend more time with family and friends. Or get a good night’s sleep. The hours have to come from somewhere, so when someone ignores the fact that I get the same 24-hour-day as everyone else, it makes me angry — especially so when it’s coming from people with better job security and higher salaries. Please stop wasting my time.
But enough about poor old me. Let’s think of the student. Sure, maybe they’ll get this through peer review and they’ll have a “nice publication” out of it. But at what cost? I finished my PhD 10 years ago and was lucky to have a good dissertation advisor and committee. They knew what they were talking about, and more importantly, they would tell me when I didn’t. I trusted them to say “Darren, fix this mess” and then help me do just that. I can’t imagine them letting me publish garbage, risking my own nascent, fragile scientific reputation in the process. Especially not in this age of post-publication review and wider discussion of research papers. I mean, what’s going to happen when this person is on the job market for the first time and someone wants to discuss their recent papers? What if it’s someone who is as sick of this stuff as I am? Risky.
And finally, we have to think about Science. Because we all know that this paper will be published somewhere, probably as is — since I doubt they will take my review on board (based on the handful of times I’ve been asked, purely by happenstance, to re-review a paper after it’s rejection and re-submission to a new journal). It’s just too easy to turn it right around to another journal and roll the dice with new reviewers.
Not that it would matter. The data themselves are irredeemable. But published it will be. It will be added to the heaping pile of garbage that pollutes the scientific record, making it harder and harder for us to find the nuggets of knowledge. Even worse, it may be taken at face-value by unsuspecting researchers or decision makers. But at least someone’s 89 page CV will be 2 sentences longer.
Anyway, my rant is over. I am out of steam. So I will leave it with the smartest thing ever written about (medical) research, hands-down:
We need less research, better research, and research done for the right reasons. — Doug Altman.
Thank you for listening.