Customer Story

Data detectives fight falsification in clinical trial reporting

A forward-thinking statistical monitoring program helps ensure smooth and accurate regulatory filings

Perennial literary favorite Sherlock Holmes once offered some sage investigative advice: “It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” While Holmes is best known for applying this tenet to an array of beguiling Victorian scandals, it is no less relevant today. In some ways, the twisting of facts is human nature; not only for the detectives investigating instances of misconduct but also for those committing such misdemeanors themselves.  

Regulatory submissions subject clinical data and analyses to intense scrutiny

In the world of drug development, as in criminology, data integrity is a serious affair. Stringent regulations put in place by regulatory agencies like the EMA and FDA are designed to protect patients from drugs that could do them harm. The onus thus falls to pharmaceutical companies to provide statistically sound clinical trial evidence prior to the release of new therapies. To ensure a smooth drug submission and approval process, the more innovative among them – companies like Roche, for example – have adopted new best practices for statistical monitoring.

A robust monitoring program is so much more than traditional data cleaning, says Chris Wells, Statistician and Team Lead for Roche’s Centralized Statistics Monitoring Group. “We need to preserve the quality and integrity of our data by ensuring that we are able to identify any occurrences of fraud or falsification of data, any calibration or training issues within and/or across [clinical trial] sites, any severe inliers or outliers or any other issue that may affect quality or put the program at risk,” she says.

“We’re looking at every Roche study with a view to data integrity. We need to make sure our data is standard, of a high level globally. And that results are true and accurate and reflect what patients are actually experiencing.” Ultimately, Wells says, better data and higher-quality submissions improve the likelihood that a new drug will be approved and made available to patients faster.

Data integrity applications in JMP^® Clinical use algorithms to identify site and patient misconduct

To achieve the best possible data quality across its many global clinical trials, Roche has pioneered next-generation statistical monitoring practices, for which JMP Clinical is seen as an essential tool. JMP Clinical not only identifies simple data anomalies (e.g., missing information), it also applies advanced statistical algorithms to a clinical data set to help uncover outliers or non-random errors that instantiate possible misconduct.

JMP Clinical enables Wells and her team to look for dubious similarities or duplicates in demographic distributions and birthdays, cluster subjects both within and across study sites, unlikely weekday or holiday dosing schedules, perfect schedules of attendance, non-variable findings, duplicate records, digit preference (a comparison across sites that helps identify data quality issues), and multivariate inliers and outliers. Specialty statistical tests in JMP Clinical, Wells says, flag potentially problematic data patterns that might not otherwise be noticed with non-statistical evaluation methods.

“Once we look at all these tests, JMP Clinical gives us flags. We then need to drill down into the data and try to understand what is happening,” says Wells. “We then refer the findings from these flags back to the study teams to review. We’ll look at patient notes and site practices to try to establish whether there are changes that need to be made.

“JMP Clinical makes you more like a detective. You have to be like Sherlock Holmes. What might a person do that could affect your data? JMP Clinical has the ability to do that. You’re only restricted by your own mind.”

A fruitful partnership with Prism helps Roche unlock the full potential of JMP^® Clinical

As every good detective needs a partner, so Wells turned to Prism Training & Consultancy, an experienced statistics company based in Cambridge, UK, with a long history of providing support to SMEs and major pharmaceutical companies globally. Three consultants from the Prism team, each with years of experience supporting clinical trials, joined Wells and her colleagues in a partnership that allowed Roche to supplement its centralized in-house team with Prism’s expertise in providing enablement training and analysis of study data using JMP Clinical for the short term.  

“Prism are one of the preferred suppliers of training based in Cambridge in the UK who specialize in enabling companies in the use of JMP Clinical. They have proven to be invaluable in the support they have provided Roche on this journey,” explains Wells.

That strategy, however, changed in the longer term, and Roche began the process to identify an external vendor to outsource the required tasks. During this period, Prism adopted an analysis and enablement role, in conjunction with Roche. And now that an outsource partner has been identified, Prism consultants are working closely with both the vendor and Roche to help train the incoming analysts in their new roles. Once the vendor team is fully active, Prism will step back to adopt a responsive training and ongoing support role, as required, which will include scripting enhancements and add-ins to the system identified by the in-house team.

Statistical tests flag issues that require further investigation

By flagging problematic clinical data before it is included in a regulatory submission, Wells’ team has a chance to take corrective action. In the vast majority of cases, Wells says, falsification is not maliciously motivated. In such instances, flagging the problem early helps clinical trial sites to correct their reporting practices for both ongoing and future studies. “Teams were very precious about their data at first,” Wells recalls. “We had to fight to get the data from them. But now teams are really wanting to give us their data ... we’ve had positive verbal feedback. The more we can do and the more we can learn from JMP Clinical, the more it will be beneficial overall.”

Data falsification is a complex issue, with countless motivations – both intentional and unintentional –  driving clinical administrators and patients around the world to report any data they suspect may be questionable. While it doesn’t happen often, on-site errors such as mixing up patient files or confusing respondents can still have a skewing impact on study outcomes. Once clinical administrators are made aware of these errors, they can put local quality checks into place.

Administrators are not the only culprits of falsification; patients can also engage in misconduct that affects clinical data integrity. For example, there are instances where patients have been identified as enrolling in multiple studies at once, whether for increased access to medical care or for financial gain. Furthermore, cultural or religious beliefs may motivate patients to inaccurately report symptoms.

JMP^® Clinical offers peace of mind paired with significant savings

“Statistical monitoring is a relatively new field,” Wells says. “It allows us to look at the data in a different way, rather than looking for just outliers, statistical algorithms help to detect noticeable trends.” Using a tool like JMP Clinical that automates much of the process allows her to make sure monitoring practices are consistent across Roche’s global studies.

The long-term benefits of consistent, high-quality reporting and drug submissions are clear, Wells says. For one, “you’ve got peace of mind when your study goes to filing with the FDA or EMA that you know what’s in your data. Generally, you’d look at data in a different way. But with [JMP Clinical], we can look across the study data and identify issues that might not necessarily be observed using previous methods. Moreover, Wells says Roche isn’t the only one using JMP Clinical; the FDA has licenses too. By preparing reports in a translatable format, she can ensure that results are as transparent as possible. Furthermore, JMP graphics help communicate advanced analyses.

Second, while findings to date have been minimal, costly delay may be averted in the regulatory review process when adopting new statistical monitoring practices. Wells estimates that should delays occur, they could cost as much as $6 million to $15 million per drug per day – no small figure even for a company with as sizeable an R&D budget as Roche. Moreover, airtight drug submissions win Roche first-to-market advantages like patents and exclusivity licenses that have long-lasting financial implications.