Lessons on standards from Clinical Trials

This week, the UK published new guidelines on Clinical trials in the field of medicine. As a field, they are ahead in terms of oversight and monitoring processes, and there are lessons to be learnt from this new document.

Are most published academic findings really false?

The issue of publication bias reared its head around the turn of the century. In 2005, John Ioannidis published a controversial paper called ‘Why Most Published Research Findings are False’ available here. Pharmaceutical companies and many others had noted that many of the findings in published academic papers were impossible to re-create in their labs. Was it possible that academics, people we trust to objectively sift through input data, have been fudging their results for career advancement?

Ioannidis’ article is a mathematical explanation of why clinical trials were producing so many seemingly spurious results. Instead of blindly assuming that a p-value of less than 0.05 was evidence of confidence, he looked at the chances of any given study with a positive finding (in other words one that has reported a statistically significant correlation between variables) is actually valid. Put simply, in any set of 100 studies with positive outcomes, 5 of those will be false positives. Add to that, the problem that:

1. A p-value of 0.05 is a minimum requirement to get published, so the many, many studies that produce results with higher p-values will be filtered into the trash can
2. Publications want surprising content, not additions to known content.

The chances of those 5 false positives ending up getting published are very high. The 95 other studies that found counter evidence won’t leave the researchers office. It is a problem of publication bias and misplaced confidence in statistics.

How did the Federal Drug Agency in the US respond to the findings?

Following a major review in the USA, the Federal Drugs Agency looked into the problem in the use of anti-depressant drugs and found trusting the publication process alone was an insufficient and flawed process. It is now required that ALL researchers carrying out clinical trials on medicines report their findings. This provides pharmaceutical companies and those in charge of life-or-death decision making complete data on which to base their decisions.

How do the new guidelines help my field?

Fast forward to this week and new best practice guidelines have been released, and serve as a guideline for all of us. The tone of the document  available here, is that it is the duty of those carrying out data initiatives to prove that they are trustworthy, putting the burden of trust on data analysts to be transparent and responsive in their actions. The document lays out steps at each stage of the process, where transparency is needed. Lead National Data Guardian for health, Dr Nicola Burne on launching the new approach has argued that it is not enough today for organisations to state that they have met minimal legal obligations. Those seeking respect for their work should be more open in sharing their processes and ready to defend their practices.

Support in this from the IoA

We’ve been concerned about transparency and trustworthy analytics here at the IoA too! If you have questions about current reporting best practices for your own data processes, check our Governance and Professionalism courses here, for some guidelines.