Harvard Prof On Stats Shifts
February 07, 2008
Not long ago, on a bleak winter morning when the U.S. Congress and the New York governor and the Connecticut attorney general and the second most important medical journal were trashing the industry, we banged out an email to Scott Evans.
He’s a senior research scientist in Harvard University’s School of Public Health. He teaches biostatistics there, is an advisor to the FDA, and the lead investigator on a few large NIH projects.
More than a year ago, Evans authored this cogent peer-reviewed article (it’s also in PDF format) about when it’s appropriate to change primary end points in clinical trials. His analysis remains timely and cogent today, with some sponsors appearing to struggle with the question.
A sample quotation from the article: “Revisions to end points (particularly primary end points) should be uncommon. If not appropriately evaluated, such revisions lead to misguided research and suboptimal patient care.”
Faith in The Blind
For starters, let’s review one thing. End points, Evans says, should not be tinkered with lightly. They’re not like the “scan” button on the radio in your car.
“Changing end points in a general sense is not advised,” says Evans, who is the principal investigator of the NIH-funded Statistical and Data Management Center for the Neurologic AIDS Research Consortium (NARC). “It can compromise the integrity of a trial. Theoretically, it can be done without compromising integrity. But it has to be done under strict conditions and looked upon as an exception to the rule.”
So end points can never be changed? No, says Evans: “There are circumstances where it can be done and is acceptable.”
A Red Flag
One appropriate such event, he said, is if new scientific information arises outside the context of the trial. Perhaps there’s a new biomarker that emerges in the scientific literature, or a different clinical trial has pointed to some relevant conclusion. Perhaps the study is a very long-term project, and the protocol anticipated such results.
By the same token, he says it is never kosher to change a primary end point if the trial data have been viewed. By anyone. And that brings up a vexing operational issue. Proving the clinical data were blinded at the time of a decision to change an end point can be tough: “It’s easier to do things in theory than it is in practice,” says Evans. “Whether that is true can be difficult to show.”
Evans says data monitoring committees (DMC), for all their prominence, may not be the appropriate bodies to make decisions about changing end points. They are pretty close to the data, after all.
“DMC are in place to uphold the integrity of trials,” he says. “They protect the blind in that they may become unblinded, but that enables the people conducting the trial to remain blinded.” Evans says it might be better to have some sort of new committee that is, like a DMC, independent of the sponsor and the trial.
For all the procedures around DMC and institutional review boards (IRB) of various types, there are no standard rules around end point changes. “The fundamental procedure for going about this hasn’t been developed,” he says. “There isn’t a standard way to go about this.”
We noted that the top person at the Journal of the American Medical Association (JAMA) was recently quoted in the Wall Street Journal, charging the industry with “manipulating” data. Evans reminded us that JAMA is now requiring an independent, third-party review of biostatistical analysis that originates at the same company sponsoring the research. There is more about that policy here, at the British Medical Journal website. To put it mildly, JAMA’s policy signals high distrust of the biostatistical expertise in the sponsor community.
“That potential is always there,” Evans says of data manipulation by industry. While he is not aware of any specific incidents, and appreciates the already heavy burden of existing regulation on industry, he muses about what might be done to ensure the data are presented accurately. “It’s worth reviewing the process to figure out if we can do more to make sure manipulation doesn’t take place,” he says. Evans and his Harvard colleagues have been reflecting on whether more frequent academic review of industry research might help.
We asked Evans about the Vytorin situation, in which Merck and Schering appear to have changed a primary end point on the Enhance trial—and then changed it back after a scientific and political outcry. Evans declined making any comment, not knowing any relevant details.
Industry’s Broken Strategy
One final thought based on our interview with Prof. Evans. The idea that the pharmaceutical industry “manipulates” clinical trial statistics is highly inflammatory and will eventually affect every company in the life sciences.
This idea is spreading beyond academia. It is gaining currency as part of the online zeitgeist and the 2008 political debate. It will not be possible to change the idea using the industry’s current strategy (a term we use loosely) for dealing with journal editor road rage. The industry’s confidence in its present strategy is puzzling in light of the decline in its reputation thus far. A new strategy may be needed.d9A2t49mkex