X
Story Stream
recent articles

FDA has a data problem. It may not be the one the agency is trying to solve.

On November 12, FDA published a Federal Register request for information on access to patient-level real-world data in drug and biologic submissions. The agency wants to know who controls the records, what happens when privacy agreements prevent inspection, and how reviewers can verify findings drawn from data held by hospitals, universities, or commercial vendors. These are real obstacles, especially for small patient populations.

Better contracts and better technology would help. But neither solves a more basic problem: for some diseases, there simply aren't enough patients to run the conventional trial FDA would ordinarily expect. That is denominator collapse, and it deserves a prominent place in FDA's discussion of real-world evidence. FDA's request focuses on access to data. It should also prompt a harder question: what counts as persuasive evidence when the patient population is exceptionally small? Precision medicine is making that question unavoidable. What once appeared to be a single disease may turn out to be several disorders, each with a different molecular cause. Scientists learn more about whom to treat just as the pool of eligible trial participants shrinks.

Ten thousand patients can become several groups of a few hundred. A further genetic distinction may leave 50 eligible patients, or 20. Yesterday's plausible trial is today's impossibility. The science advanced. The arithmetic did not cooperate.

Patients facing devastating diseases deserve reliable evidence. They also deserve a regulatory system that recognizes when its usual demands cannot be met. Requiring enrollment from a population that does not exist is hardly a demonstration of scientific rigor. The important questions remain the familiar ones: Who was studied? What was measured? What might explain the results? And how much confidence should reviewers place in the answer? A larger patient population is no guarantee of better evidence.

Take a registry that has followed 40 patients with the same well-defined mutation for years. For a particular clinical question, it might prove more informative than a much larger database with uncertain diagnoses and gaps in follow-up. It might also mislead. Selection bias, missing records, and inconsistent measurements can distort a small registry badly. FDA's job is to examine the evidence that exists, with all its limitations, against the question it is being asked to answer.

Randomized trials remain the strongest protection against many forms of bias. Their importance is not diminished by denominator collapse. But randomization cannot manufacture patients who do not exist.

When only a handful of patients have the relevant disease, natural-history records may help establish the untreated course. Longitudinal observations can show whether deterioration follows a recognizable pattern. Biomarkers may add context; external controls may offer comparisons. Each source needs to be examined for what it can establish, and what it cannot.

Biomarkers and historical comparisons both demand care. A change in a laboratory measure may say little about how a patient feels or functions. And an apparent treatment effect can vanish once differences between patient groups are taken into account.

Judea Pearl's work on causal inference gets to the heart of the matter: what happened after treatment is not necessarily what happened because of it. With only a few patients, a misleading comparison can look remarkably convincing.

Consider a treated group that is younger, less severely affected, or diagnosed earlier than the patients in a natural-history registry. An apparent benefit may owe more to those differences than to the medicine. Investigators need to explain how they chose the comparison group, what assumptions they made, and whether the finding survives reasonable challenges to those assumptions. Pearl, Glymour, and Jewell explain why this matters.

Bayesian methods can help when credible prior evidence exists. They cannot rescue a poor comparison. Reviewers still have to say what the records support and where the uncertainty remains.

The problem is not confined to statistics. One university registry may hold much of the usable information on an ultra-rare disorder. Families may have supplied records for years. Yet restrictions on data transfer, privacy, or FDA access can leave those observations effectively out of reach when a therapy finally enters review.

The November notice addresses precisely this predicament. FDA must be able to examine the underlying records and meet its legal obligations. Privacy is particularly sensitive when a rare combination of clinical details can identify a patient without a name attached. Still, an excellent registry should not be stranded because no standard agreement accommodates it.

Standard data-access agreements, direct submissions by qualified third parties, and secure review environments all merit consideration. FDA must still be able to reproduce analyses and retain the records the law requires. Getting the data into reviewers' hands solves only the first part of the problem.

Suppose there are 25 patients whose disease has been carefully characterized. FDA should examine what their records actually show, without assuming that the small number settles the question in advance. That scrutiny may support a regulatory decision, or it may expose gaps that cannot be bridged. Either outcome is more useful than insisting on a trial that cannot be enrolled.

FDA's Plausible Mechanism Framework offers a way forward. As molecular discoveries divide diseases into ever-smaller populations, an understanding of the biology can help reviewers interpret clinical observations that cannot be tested in a conventional trial. A plausible mechanism cannot establish clinical effectiveness on its own. It may, however, help reviewers make sense of limited clinical findings.

A plausible mechanism becomes more persuasive when clinical observations fit the biology and a fair comparison supports the same interpretation. Reviewers should explain where that chain of reasoning holds and where it breaks.

For a common disease, an unsuccessful trial may be followed by another study, another sponsor, or another therapeutic candidate. Patients with ultra-rare conditions often have no such luxury. There may be only one serious development program, one available registry, and a small number of patients who can participate.

A failed trial design can consume much of the available population. Years spent debating an impracticable enrollment target can exhaust the resources supporting development. Patients don't get that time back. None of this justifies approving an ineffective or unsafe medicine. But neither does patient protection require pretending that the arithmetic of an ultra-rare disease is identical to that of hypertension or diabetes. FDA can address this before the gap grows wider.

FDA should use its November request to set clearer expectations for evidence drawn from very small patient populations.[1] Natural-history studies, external controls, and causal analyses may inform regulatory decisions, but sponsors need to know early what would make those approaches credible and what would rule them out. Those conversations belong at the start of development, before years and resources are committed to enrollment targets that cannot be met. The expectations should be clear without locking sponsors into an evidence plan that new science may render obsolete.

Precision medicine will keep splitting familiar diagnoses into smaller groups. That is what better biology does. The regulatory consequences cannot be treated as an afterthought, especially when a development program may depend on a few dozen patients.

FDA is right to ask whether reviewers can inspect the underlying records. The harder question is what those records can actually tell them. Counting patients is easy. Judging the evidence takes more work.

FDA cannot change the arithmetic of rare disease. It can change how it responds to it. That is where regulatory science must catch up with medical science.

Peter J. Pitts, a former FDA Associate Commissioner, is President of the Center for Medicine in the Public Interest.

Comment
Show comments Hide Comments