By McKenzie E. Cato* and Jeff N. Gibbs –
As we reported previously (see previous post here), FDA has issued two long-awaited draft guidances on next generation sequencing (NGS) diagnostic tests. The first, “Use of Standards in FDA Regulatory Oversight of Next Generation Sequencing (NGS)-Based In Vitro Diagnostics (IVDs) Used for Diagnosing Germline Diseases,” provides recommendations for designing, developing, and validating NGS in vitro diagnostics (IVDs) and discusses the use of standards as part of the regulatory controls. The second, “Use of Public Human Genetic Variant Databases to Support Clinical Validity for Next Generation Sequencing (NGS)-Based In Vitro Diagnostics,” describes how publicly available databases can be used as a source of valid scientific evidence to support the clinical validity of an NGS IVD.
These topics have been extensively discussed already. NGS is playing an increasingly prominent role in research and diagnosis. Prior to releasing these draft guidances, FDA held three public meetings and released a discussion paper on regulation of and standards relating to NGS IVDs (see previous post). Further efforts by FDA on this front include the creation of “precisionFDA,” an online research and development portal “that allows for testing, piloting, and validating existing and new bioinformatics approaches to NGS processing.”
The key features of each draft guidance are outlined below.
Use of Standards in FDA Regulatory Oversight of Next Generation Sequencing (NGS)-Based In Vitro Diagnostics (IVDs) Used for Diagnosing Germline Diseases
The scope of this draft guidance is limited to targeted and whole exome human DNA sequencing (WES) NGS-based tests for suspected germline diseases. The recommendations are not intended to apply to NGS-based tests for stand-alone diagnostic purposes, screening, microbial genome testing, risk prediction, cell-free DNA testing, fetal testing, pre-implantation embryo testing, tumor genome sequencing, RNA sequencing, or use as companion diagnostics. In other words, the proposal covers only a small subset of potential applications. Furthermore, this draft guidance addresses only analytical validity, not clinical validity. Both analytical and clinical validity are required to demonstrate a reasonable assurance of safety and effectiveness of an NGS-based test. (FDA has also just issued a draft guidance on co-development of companion diagnostics which stresses this point for diagnostics.)
The few cleared NGS-based tests are single-gene, disease-specific, and targeted. Because FDA has not yet classified NGS-based tests with a broad intended use for suspected germline diseases, they are automatically classified into class III as a matter of law. However, FDA notes that it believes an NGS-based test for germline disease would be a suitable candidate for de novo classification, and encourages applicants to use the pre-submission process to discuss anticipated de novo requests. Since FDA cannot guarantee that any particular device will be granted de novo status, this is about as strong endorsement as FDA can give of the de novo pathway. The draft guidance states that if FDA were to classify NGS-based tests as class II in response to a de novo request, it would consider exempting the tests from premarket notification requirements. The factors FDA considers in determining whether a class II device may be exempt from premarket notification are outlined in a 1998 guidance. Once finalized, this guidance will supersede the 1998 guidance with respect to NGS-based tests for germline diseases.
FDA explains in the draft guidance that one way to show the analytical validation of an NGS-based test is through conformity to FDA-recognized standards or special controls. In order for a standard to be recognized by FDA, it should at a minimum include the design, development, and validation activities outlined in this draft guidance.
Test Design Considerations:
Satisfying FDA’s data requirements will not be a trivial task. Test developers are advised to prospectively define and document the indications for use statement, the features necessary to address specific user needs, the acceptable specimen types, and the region(s) of the genome that will be tested. To demonstrate performance, FDA recommends that test developers define and document the test metrics, performance thresholds for the metrics, the degree to which interrogated regions that do not meet test run quality metrics can be included, the use of secondary procedures, and possible limitations to test performance.
In addition, FDA recommends that developers specify and document all test components, set technical specifications for each step of an NGS-based test, and document the limitations of each component for critical factors. The draft guidance identifies several specific test design considerations for the sequencing platform, controls and reference materials, and bioinformatics components.
Developers are advised to develop and document methods for each step of the test; procedures for using instruments, consumables, reagents, and supporting methods; any limitations for each step; and the type of sequencing that will be used. The draft guidance includes specific recommendations for sample preparation and input, multiplexing (i.e., simultaneous testing of multiple samples), library preparation and target enrichment, and follow-up procedures to be used when a test run fails.
Test Performance Characteristics: The draft guidance recommends the following performance metrics for analytical validation of NGS-based tests for germline diseases:
- Accuracy: Demonstrate accuracy by measuring positive percent agreement, negative percent agreement, technical positive predictive value, and the rate of “no calls” or “invalid calls.”
- Precision: Evaluate precision (i.e., reproducibility and repeatability) for both variant and wild type calls.
- Limit of Detection: Establish and document the minimum and maximum amount of DNA that will enable the test to provide expected results in 95% of test runs with an acceptable rate of “no calls” or “invalid calls.”
- Analytical Specificity: Document any interfering substances, the potential for cross-reactivity of known cross-reactive alleles and homologous regions, and methods to detect cross-contamination between samples.
Test Run Quality Metrics: Test developers are advised to establish and document performance thresholds for average and minimum depths of coverage, uniformity of coverage, and percentage of bases in the target region(s) above the minimum depth of coverage for the test. The draft guidance states that FDA will not recommend specific thresholds for coverage metrics. However, FDA indicates that thresholds should not be set below the levels specified in the draft guidance.
The draft guidance includes a list of factors for establishing test run metrics and performance thresholds for the following test elements: specimen quality, DNA quality and processing, sequence generation and base-calling, mapping or assembly metrics, and variant calling metrics.
General Recommendations for Performance Evaluation Studies: The draft guidance includes an extensive list of features that FDA recommends incorporating into performance evaluation studies. This list includes recommendations regarding performance test design and configuration, recommendations for evaluating NGS-based test accuracy, and recommendations for documenting the results of validation studies. If test developers deviate from, delete, or add to these recommendations, FDA requests that they provide a “detailed justification.” As always, companies that deviate from the “guidance” should do so thoughtfully and for non-trivial reasons.
Supplemental Procedures: The draft guidance recommends including any applicable supplemental procedures (i.e., procedures that are not a part of the main process for generating variant calls from input specimens) whose reflex use will be directed in the test’s instructions in design, development and validation activities, and documentation.
Variant Annotation and Filtering: Test developers are advised to select filtering algorithms appropriate for the indications for use, establish and document filtering thresholds, and document how and when filtering will be used. The draft guidance recommends documenting any filtering criteria and their purpose.
Presentation of Test Performance: The draft guidance provides a list of recommendations for the presentation of test performance and design. Of note, the draft guidance recommends providing public access to the test’s indications for use statement, limitations, and summary performance information “via a prominent link on the company website.” This, of course, is not a typical expectation for a device.
Test Reports: The draft guidance recommends that the following information be included in test reports: the relationship between reported variants and the clinical presentation of the patient, a description of genomic and chromosomal regions detected by the test, a summary of the results of performance studies, a “prominently-placed” list of pathogenic or actionable variants on the first page of the report, test limitations, and risk mitigation elements. One of the recurring issues in the Laboratory Developed Test debate has been the extent to which FDA could regulate test reports.
Modifications: Finally, the draft guidance addresses modifications to NGS-based tests. It states that, in order to stay within the scope of the guidance, any modifications to targeted and WES NGS-based tests should retain the intended use of aiding in the diagnosis of suspected germline diseases. FDA offers the following recommendations for sponsors when making modifications: document all modifications to the test; prepare a detailed standard operating procedure (SOP) for revalidation; conduct revalidation using a sufficient number of well-characterized samples; document the types of validation studies that will be conducted after modification; where appropriate, revalidate the test end-to-end, not just the modification; assess each modification separately and in aggregate; when adding new genes to an existing panel, evaluate test performance for the original genes on the panel; and include a procedure to account for updates to internal and external databases and their potential impact on clinical interpretation of variants. Undoubtedly questions will arise over the regulatory obligations, including the need for a new submission, when FDA germline tests are modified – which they will be.
Use of Public Human Genetic Variant Databases to Support Clinical Validity for Next Generation Sequencing (NGS)-Based In Vitro Diagnostics
Unlike the other draft guidance, this draft guidance addresses clinical validity, albeit in one specific manner. It describes FDA’s considerations in determining whether a publicly available genetic variant database can serve as a source of valid scientific evidence to support the clinical validity of an NGS-based test in a premarket submission and the process by which administrators of such databases can voluntarily apply to FDA for recognition. The draft guidance notes that, depending on the sufficiency of the evidence from genetic variant databases, FDA may determine that the submission of additional valid scientific evidence is not necessary. That, of course, would substantially facilitate an application that could use a database. The devil, though, is in the details.
Recommendations to Support Recognition of Genetic Variant Databases as Sources of Valid Scientific Evidence Supporting Clinical Validity of NGS Tests:
FDA believes that a genetic variant database which could serve as a source of valid scientific evidence to support clinical validity of a test would have the following characteristics:
- Provide sufficient assurances regarding the quality of the source data;
- Provide transparency regarding its data sources and operations;
- Collect, store, and report data in compliance with all requirements regarding protected health information, patient privacy, research subject protections, and data security; and
- House sequence information generated by validated methods.
The draft guidance recommends that genetic variant databases make sufficient information regarding data sources and SOPs publicly available. Asking that SOPs be made publicly available is unusual, and may not be well-received. The draft guidance further states that the SOPs should define how variant information is aggregated, curated, and interpreted. Database administrators are advised to have processes in place for assessing database stability and to back up the database on a regular basis. Additionally, the draft guidance recommends that database administrators use commonly accepted data formats.
To promote data quality, the draft guidance recommends that databases use consistent nomenclature that is widely accepted by the genomics community. The draft guidance also states that all variant data should be accompanied by metadata, including the number of laboratories or studies reporting the variant classification, the names of the laboratories that reported the variant, the name of the test used to detect the variant, variant characteristics, and details of the technical characteristics of the test used. The draft guidance recommends that database operations include methods to ensure that individual data points are not represented more than once.
The draft guidance recommends that all SOPs for curation and variant interpretation be publicly available. Additionally, FDA recommends that the SOPS include validated decisions matrices, such as those based on well-recognized professional guidelines. FDA recommends that databases define the types of evidence that personnel interpreting variants may use for an interpretation.
The draft guidance states that assertions (i.e., assessments of a genotype-phenotype correlation for a particular variant) should be adequately supported and appropriate for the level of certainty and nature of the genotype-phenotype relationship. However, the draft guidance cautions that in order to be FDA-recognized, a genetic variant database should not include and recommendations regarding clinical treatment or diagnosis.
Database administrators are advised to ensure that all personnel who are interpreting variants have adequate training and expertise. The draft guidance also recommends that there be methods in place, such as proficiency testing, to make sure that personnel continue to maintain high quality standards over time. In order to be considered for FDA-recognition, database administrators should make efforts to minimize and make transparent any potential conflicts of interest regarding the database or its personnel. FDA’s proposal to provide guidance to database administrators represents an intriguing potential expansion of FDA’s reach to a new class of entities.
Genetic Variant Database Recognition Process:
The draft guidance states that FDA intends for the recognition process to involve three steps: (1) voluntary submission of detailed information about the database; (2) FDA review of genetic variant database policies and procedures for obtaining and maintaining data and making variant assertions; and (3) maintenance of FDA recognition of a database.
Database administrators who are seeking to have their variant assertions recognized by FDA as valid scientific evidence to support clinical validity of NGS-based tests can make a voluntary submission to FDA which demonstrates that the database conforms to the recommendations in the guidance. The draft guidance includes a list of the types of documents which can be included in this submission (e.g., policies, data preservation plans, validation studies for interpretation SOPs).
The draft guidance states that, where appropriate, FDA may verify variant assertions to make sure they are supported and that the database is following its SOPs. FDA will treat these voluntary submissions as confidential prior to recognition. FDA intends to post on its website a list of all FDA-recognized genetic variant databases “and other relevant, public information about those databases.”
FDA stated that it plans to review FDA-recognized databases regularly to verify that they continue to follow their SOPs and the recommendations in this guidance. FDA will consider the following factors when evaluating whether a genetic variant database for NGS-based tests may maintain FDA recognition:
- Processes should incorporate multiple lines of scientific evidence with appropriate weights.
- Processes should use a tiered system of assertions and adequately describe the meanings of each tier.
- Databases should implement a decision matrix based on validated SOPs or rigorous professional guidelines that incorporate details of the gene or disease evaluated.
- Databases should include validations of the decision matrix.
- Guidelines, decision matrices, and details supporting each variant’s interpretation should be publicly available.
FDA does not discuss what happens to applicants who are in the midst of the review process of the database they are relying upon to keep its recognition.
The draft guidance notes that FDA may consider using third parties to assist with genetic variant database recognition in the future, as it does with the third party 510(k) review program. This, too, is intriguing, because it would entail use of third party review beyond the statutorily enumerated categories.
In theory, the ability to use curated databases in lieu of clinical validity studies is appealing to would-be applicants. A separate question is whether undergoing FDA scrutiny will be sufficiently appealing to database administrators.
* Summer Associate