The 2008 Drug Information Association (DIA) annual meeting featured a painful session on metrics in clinical trials. Not painful because of any of the speakers, who were polished and insightful.

No, it was painful because sizable clinical research projects and sophisticated companies are not really being managed. We’re using the word “managed” in the sense that other industries manage time, resources and people.

Indeed, all three speakers had examples of metrics that produced diametrically opposite results to what was probably good for the company in question or originally intended by the people who came up with the metrics. And in fairness, many industries have struggled with choosing the right metrics. Pharma is not alone.

All of the speakers generally conceded that in the life sciences, what is being counted (not to mention the the way it’s being tracked) is unlikely to be in harmony with what senior management desires.

Long Flights

Eric Lake is a partner at Pharmica Consulting, a New Jersey-based consultancy. He shared a few anecdotes from other industries that were illuminating. Southwest Airlines decided to try to save on its fuel bill by tracking the fuel usage of each pilot, with appropriate incentives.

Pilots elected to fly slower, run the air conditioning less, and generally save as much energy as possible. The result? Lower fuel bills. But also very dissatisfied business travelers.

In pharma, Lake said, there is often a reflexive desire to gather data for its own sake. That may introduce inefficiencies. Some data is collected just so that it can sit on a shelf. “Everything is about reports,” he said. “If you have metrics and you’re not reporting on them, I don’t know what you’re doing. The reports are what it’s all about.”

TMI

Lake also suggested that the pharmaceutical industry gathers far too much data. “This is the one everybody falls in,” Lake says. “No one really knows what is important and what is not.” General Motors had a similar problem, he noted, once collecting 600 or 700 indications of quality. The auto firm eventually slashed that number. “No one should be looking at 6,000 metrics, nor 600 nor 60,” he said.

Even more disconcertingly, he said, some data are being gathered for appearance’s sake, not because any decision or change will be made because of the metric. “Any metric should be actionable,” he said.

Forward-Looking

Finally, Lake noted, metrics in clinical trials can be retrospective in ways that are counterproductive. “Ideal metrics should provide leading indicators,” he said.

Carol Seider, associate director of field operations at Merck, took an equally tough line. “I challenge you to find any company in our industry using metrics correctly,” she said at the start of her presentation. She assesses clinical research associate performance at the company.

The issue is not whether metrics are a good idea in principle. “How do we choose the right very few metrics to measure a site’s and CRA’s performance?” she asked.

What To Count

It was clear in her talk that different companies will have unique approaches to winnowing down a long list of potential metrics. Potentially empty metrics in her experience, she suggested, might include: days until first patient visit; number of CRA visits to clinical sites; number of sites for which source documents have been verified; and hours spent doing site validation.

Merck subsequently decided to use other metrics: whether a site met its enrollment target; whether the data were accurate and punctual; protocol violations; and audit findings. 

At Genzyme, meanwhile, Randy Krauss is encountering similar issues. Associate director of biomedical regulatory affairs, Krauss said that one attempt to list what could be measured at his company turned into a months long, nearly interminable exercise. “There is no shortage of what to measure,” Krauss said. “There is no limit to the number of metrics you could capture.”

EVA Tools

Faced with limited data in the public domain, Genzyme began working with CMR International, a Thomson-Reuters division, to get some industrywide perspective. Krauss also seems pleased with a bit of MBA wizardry called an earned value analysis (EVA).

It can be calculated manually, or by Microsoft Project. EVA measures project performance and provides an early warning of projects that are about to veer off track or over budget. You can find a simple EVA calculator online here.

In one scenario, Krauss used EVA to predict a project that had already been completed at the company. In actuality, the project took 7.6 months, not the expected 3.3 months. The cost was several million dollars more than anticipated. The EVA technique came significantly closer to predicting the actual completion date and budget than a simplistic human estimate.

Krauss was an understated but effective advocate for metrics in a buzzword-free manner. “Metrics should be used in group planning to identify areas of concern,” he said.

Metrics 101

The session was replete with jaw-dropping examples. Most but not all were from the life sciences. One firm hires clinical research associates as independent contractors, paying them twice what would have been necessary to bring them aboard as fulltime employees. (The CRAs were used on a longterm basis, not for short bursts of activity.) What was the economic justification? The funding came out of a headquarters budget, not a local budget.

Deliberately or not, the session displayed relatively unsophisticated industry usage of metrics and, more broadly, of time and human capital in the clinical trial sponsor community. Perhaps next year a few contract research organizations will be able to assess the same topic, and bare their metric-related experiences to the same degree.