GAO Issues Report on Non-Inferiority Studies; Finds No Evidence of “Biocreep,” But More Conservative Use of Such Studies to Support Approval

By Kurt R. Karst –   

Responding to a request from certain Members of Congress, the U.S. Government Accountability Office (“GAO”) has released a report evaluating FDA’s use of evidence from non-inferiority trials to establish a drug’s effectiveness and support approval of new therapies.  As we previously reported when FDA issued its “Guidance for Industry: Non-Inferiority Clinical Trials” in March 2010, a non-inferiority trial seeks to demonstrate that “any difference between [ ] two treatments is small enough to allow a conclusion that the new drug has at least some effect or, in many cases, an effect that is not too much smaller than the active control.”  This is in contrast to the more common superiority trials, such as a placebo-controlled trial, which seek to prove a new drug is more effective than the control. 

Non-inferiority trials are most often used when it would be unethical to use a placebo control.  As the GAO report points out, however, there are certain issues unique to non-inferiority studies that have raised concern over the years, including that “[u]sing data from the non-inferiority trial and from prior trials measuring the effectiveness of the active control, the effectiveness of the new drug is estimated – but not ever fully known.”  In addition, “non-inferiority trials are more prone to certain biases than superiority trials,” and the “use of non-inferiority trials over time also raises concerns about the potential for ‘biocreep’ to occur” – that is “the concern that successive generations of drugs approved based on non-inferiority trials, with the active control changing in each new generation, could lead to the adoption of decreasingly effective drugs and ultimately to the approval of drugs that are no more effective than a placebo.”

The GAO report, titled “New Drug Approval: FDA's Consideration of Evidence from Certain Clinical Trials,” evaluates FDA’s use of evidence from non-inferiority trials using a three-step approach: “(1) identify the type and status of drug applications submitted for FDA review that included evidence from non-inferiority trials; (2) examine the characteristics of non-inferiority trials FDA considered in making approval decisions; and (3) describe FDA’s guidance for establishing a drug’s effectiveness on the basis of non-inferiority trials.”

The GAO report winnows down the number of NDA’s submitted to FDA between Fiscal Years 2002 and 2009 (October 1, 2001, through September 30, 2009) to 175 NDAs for new molecular entities – 43 (or one quarter of which) included evidence from at least one non-inferiority study.  Of the 43 NDAs identified, 29 of them included evidence from at least one non-inferiority trial.  “Most NDAs – 18 of the 29 – were approved based on evidence from pivotal non-inferiority trials.  FDA approved the remaining 11 applications based on other evidence, such as the superiority of the new drug compared to a placebo or an active control.”  Twelve of the 18 NDAs approved based on evidence from pivotal non-inferiority trials were for antimicrobial drugs.  Interestingly, the GAO report notes that “[t]he number of NDAs with evidence from non-inferiority trials varied from year to year and generally declined from fiscal years 2002 through 2009.  On average, FDA received five NDAs each year that included evidence from non-inferiority trials.”

With respect to non-inferiority study characteristics, the GAO report notes that:

Characteristics varied among the non-inferiority trials that provided primary evidence for the approval of the 18 NDAs.  FDA relied on primary evidence from multiple pivotal non-inferiority trials to support the approval of most of these applications. The number of pivotal non-inferiority trials used as primary evidence for these 18 NDAs ranged from one to four, with an average of two pivotal non-inferiority trials supporting the approval of each application.

Some other interesting tidbits from the report include:

• “Two-thirds, or 12, of the 18 NDAs included trials that measured drug effectiveness using a surrogate, rather than a clinical, primary endpoint in at least one of their pivotal trials.”  According to some experts, this “increases uncertainty in the drugs’ true effectiveness.”

• “Half of the 18 NDAs FDA approved on the basis of non-inferiority trials tested the effectiveness of the new drug against more than one active control.  A majority of the active controls used in non-inferiority trials were FDA-approved for the indication. However, three applications included evidence from non-inferiority trials that used one active control that was not FDA-approved for the indication.”

• “The margins used for most of the 18 NDAs approved on the basis of evidence from non-inferiority trials ranged from 5 to 20 percent, with the most commonly used margin being 10 percent.  That is, for trials using a 10 percent non-inferiority margin, the new drug could be estimated to be up to 10 percent less effective than the active control. However, the observed difference in the effectiveness of the new drug and active control, as measured in the clinical trials, would be less than 10 percent.”

• “FDA did not agree with the non-inferiority margins set for pivotal trials submitted with three applications [i.e., NOXAFIL, EXJADE, and REYATAZ)], though the agency approved these drugs based on evidence from these trials.

• “FDA reviewed the characteristics of the non-inferiority trials supporting the approval of the 18 NDAs to ensure that the drugs it approved were more effective than a placebo. FDA’s review therefore minimized the potential for biocreep. Similarly, our examination of the trials’ characteristics also revealed no evidence of biocreep.”

• “While non-inferiority trials provided primary evidence of effectiveness to support the approval of 18 NDAs, other non-inferiority trials were poorly designed and did not provide such evidence. Of the other 25 NDAs that included evidence from non-inferiority trials, FDA identified 9 applications that included poorly designed non-inferiority trials.”

The GAO report also commented on FDA’s March 2010 guidance on non-inferiority studies.  As we previously reported, in addition to providing recommendations regarding study design and interpretation, the guidance provides answers to nine “commonly asked questions” regarding the estimation of margins, appropriate active control drugs, endpoints, and reliance on a single non-inferiority study to support effectiveness.  The guidance also discusses five examples derived from publicly available information that describe how to choose a non-inferiority margin, how to analyze the results, and other considerations relevant to the design and interpretation of non-inferiority studies.  According to the GAO report, “[w]hile experts we interviewed who reviewed FDA’s March 2010 guidance noted that it addressed key principles, most identified additional technical issues that they would have liked this guidance to have addressed.”  Such additional issues include “how the use of a surrogate endpoint impacts the design and interpretation of a non-inferiority trial,” and the need for more “detailed instructions on how to estimate the effect of the active control in the non-inferiority trial.”  

Finally, the GAO report concludes with a general observation that FDA “has become more conservative in allowing evidence from non-inferiority trials to demonstrate the effectiveness of new drugs” – perhaps explaining their general decline in recent years.  According to the GAO, two points support this cinclusion:

First, FDA has revised its view regarding when non-inferiority trials may be used. Prior to 2007, for example, FDA had approved drugs treating several less severe infections – including acute bacterial sinusitis, acute bacterial otitis media, and acute bacterial exacerbations of chronic bronchitis – on the basis of evidence from non-inferiority trials. . . .  Second, FDA has become more rigorous in its review of evidence from non-inferiority trials.  For example, prior to 2001, FDA’s guidance on the development of anti-infective drugs had not advised sponsors to scientifically calculate or justify their selected non-inferiority margins – a step that FDA’s March 2010 guidance recommends.

Other than technical comments, FDA did not respond to the GAO report.  Thus, FDA would apparently agree with the GAO’s view that the Agency is becoming more conservative in its acceptance of data from non-inferiority studies to support drug approval.