A scientist’s perspective on scientific evidence.


When I started as a scientist publication was all, it still is. I now understand though that if I write a paper, it will be published. Maybe not in my first choice of journal, but however good or bad it is, there is somewhere that will publish it. Now journals accept papers on the same day or the day after submission: (bit.ly/1KWaV2c). Do we really think they’re getting peer reviewed? When I want to know more about a condition, I don’t just Google something I look at the academic literature, try and find out what the evidence suggests, but is that also just someone’s opinion now?

When I was a young, naive PhD student I was slightly shocked that journals asked me to suggest my own reviewers. Not that naive, I just named the people I was currently flirting with, but still it did seem to lack a little subjectivity. Fast-forward a decade or two and the profit-based world of publication has exploded.  Editors need to get papers in and out again through the pipeline. Authors need to get those manuscripts off their desks and on their CVs and grant reports. Quantity over quality every time. I have now written papers which are essentially just my opinion, waffling away, and they’re published not in journals on discourse but scientific journals that lend my thoughts and air of peer-reviewed authority. These are no longer my thoughts, these are now evidence-based facts.

So, if you’re reading up on a clinical condition, what should you look for?

1. Randomized Controlled Trials (RCT)

Just having collected some data and analysed some statistics doesn’t make it good evidence. If I flip a coin and it lands on heads four times in a row, is that a rare and noteworthy event? No, it’s a sign of randomness. If I flipped a coin and it neatly alternated between heads and tails, now that would be noteworthy. I once co-wrote a paper with an amazing statistician, demonstrating how the statistics in one of my previous papers were a misleading mistake. No one would publish it. Randomized controlled trials are the best we can do for looking for actual significance.

2. Cochrane reviews.

These are a special type of review of the existing papers. They collect together eligible studies into a meta-study, telling you what the balance of the evidence says. These try to include only peer-reviewed papers that have a rigorous method and so collect together a much larger, combined subject group. They use some fairly robust statistics, and are a hell of a lot more accurate than just trying to read and counter-balance all the papers yourself, by eye.

3. A bit of time for peer review

Most published manuscripts have date of submission and date of acceptance stamped on them. I used to think of this as a badge of shame: if it takes me a year to get my paper out everyone knows that. Oh, how the world changes. Look for something that’s obviously been through a  discussion process, preferably a long one, not just same day/next day acceptance. Publication bias means it’s actually much harder and slower to publish a paper with a negative result – “we looked but actually it wasn’t there”, “this doesn’t work” – or a paper that goes against the mainstream. Doesn’t make it a bad paper, on the contrary, unexpectedly negative findings can often be the important ones.

4. Big name journals.

There’s a lot to be said about impact factors and how to judge a good journal which doesn’t really belong here. Fact remains, how often a  journal is cited and how often a paper is cited says a lot about how seriously academics are taking any paper. Of course it doesn’t tell you whether people are agreeing or disagreeing with it, but any academic mention is still more reassuring than a paper that is politely ignored by the scientific community. Those lesser-known ones, in the journals with the ridiculous names, those are the ones that you need to politely ignore too. Scientists don’t actually ignore papers just because they disagree – it doesn’t take a second to write “Contrary to the findings of Blogg and Blogg (2015), we showed…” Having someone to disprove makes your paper more publishable – you’re now showing something that actually needed to be shown.

5. Conference papers

Any collection of papers that has a nice holiday attached a.k.a. conference papers, is going to get our slightly less completed thoughts. Conference papers are often published in mainstream journals and look like full papers on your CV, but depending on your field it’s fairly hard to get a conference paper rejected. They only need preliminary findings as these should be the cutting edge of your ongoing research, your chance to tell the field what you’re currently working on and get the feedback and awkward questions that’ll make the next round of tests and the final full paper a far more reliable affair. I for one am on the committee of a lot of the conferences I want to go to, and at the Keynote career stage. This means that I often get asked to submit a paper for a talk that I’ve already been asked to give. You can peer-review that all you like, it’s already been accepted.

These days scientific publication falls under the more general category of things we read on the internet  – where even if it’s an intelligent person’s best guess at the time it’s not necessarily true. That’s OK, it never was – scientists don’t believe in facts, they believe in best current theory and convergence towards a solution. Just remember that scientific papers are not the protected species that they used to be, and should be read with the scepticism and critical thinking that allows the good ones to shine.

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s