Assessing the Credibility of Computational Modeling and Simulation in Medical Device Submissions

On December 23, 2021, CDRH released as a draft guidance, Assessing the Credibility of Computational Modeling and Simulation in Medical Device Submissions (Draft Guidance).  Computational modeling and simulation (CM&S) can sometimes be useful to demonstrate the safety and effectiveness of medical devices or incorporated into devices.  FDA indicates that they receive regulatory submissions with such computational modeling, but the submissions “often lack a clear rationale for why models can be considered credible for the context of use.” Draft Guidance at 4.

The Draft Guidance describes a nine-step framework for evaluating the credibility of CM&S information submitted in pre-market applications.  A computational model is “the numerical implementation of the mathematical model performed by means of a computer.” Draft Guidance at 8.  The National Institute of Biomedical Imaging and Bioengineering describes computational modeling as “the use of computers to simulate and study complex systems using mathematics, physics and computer science.” NIH, Computational Modeling (May 2020).  Weather forecasting is an example of computational modeling and simulation for which most of us are familiar.  Similar techniques can be used to model complex biological systems. The Draft Guidance applies to physics-based or mechanistic CM&S and not statistical or data-driven CM&S, such as those incorporating artificial intelligence or machine learning.

The Draft Guidance describes four types of CM&S that can potentially be used to support a regulatory submission, by either being used to provide evidence to support a device’s safety and effectiveness or by being incorporated within the device, itself:

• In Silico Device Testing, which are computational models that simulate medical device performance. Draft Guidance at 5.
• CM&S used within medical device software, which is use of computational modeling within medical device software to perform device functions. at 6.
• In Silico Clinical Trials, where “device performance is evaluated using a ‘virtual cohort’ of simulated patients with realistic anatomical and physiological variability representing the indicated patient population.”
• CM&S-based qualified tools, which are tools for developing or evaluating a medical device that can be submitted to CDRH under the Medical Device Development Tools (MDDT) Program.

In the Draft Guidance, credibility is defined as “trust in the predictive capability of a computational model.” Id. at 4.  The guidance assesses credibility using key concepts from FDA-recognized standard ASME V&V 40 Assessing Credibility of Computational Modeling through Verification and Validation: Application to Medical Devices.  However, where the ASME standard assumes the ability to perform traditional validation activities, the Draft Guidance provides a more general framework that additionally incorporates non-traditional validation evidence.

A nine-step framework is presented for assessing credibility for purposes of a regulatory submission of the four types of computational modeling described above.  The first steps are to (1) describe the question of interest, (2) describe context of use and (3) model risk.  Next, (4) credibility evidence, either previously generated or planned, is identified and categorized, followed by (5) defining credibility factors and setting prospective credibility goals.  Prospective adequacy assessment is then performed (6) to answer the question, “will the credibility evidence be sufficient to support using the model for the context of use given the risk assessment?” Id. at 10.  Credibility evidence is then generated (7) by executing the proposed studies and/or analyzing previously generated data.  A post-study adequacy assessment (8) is conducted to determine if credibility goals were met, followed by preparation of a credibility report (9).

Key concepts from the framework are then presented in detail within the Draft Guidance, including points of consideration for each type of CM&S, where applicable.  The question of interest should describe the question that is being addressed using the model and along with other sources of information.  When considering the context of use, it should be the specific role and scope of the computational model used to address the question of interest.  The Draft Guidance recommends that a model’s risk, defined as “the possibility that the computational model and the simulation results may lead to an incorrect decision that would lead to an adverse outcome” is assessed according to the ISO 14971 and ASME V&V 40 standards.  Id. at 9.

Credibility evidence is evidence that could support the credibility of a computational model.  Id. at 8. There are three types of credibility evidence (code verification, calculation verification, validation) and ten distinct categories within these three types of credibility evidence that are discussed in the Draft Guidance.

Code verification provides evidence demonstrating that a computational model implemented in software is an accurate implementation of the underlying mathematical model.  Calculation verification determines the solution accuracy of a calculation.  Both calculation verification and validation of the model may be provided through a number of types of evidence, including: general non-context-of-use evidence, evidence generated using bench-top conductions to support the current context of use, evidence generated using in vivo conditions to support the current context of use, evidence generated using bench-top conductions to support a different context of use, and evidence generated using in vivo conditions to support a different context of use.  Validation can additionally be provided by population-based evidence, emergent model behavior, model plausibility and model calibration evidence.  Model calibration evidence is an assessment of the fit of simulation results against the data used to develop the model; while it can support the validation of the model, it alone cannot be used to validate the model.

Although a pre-submission is optional, the Draft Guidance suggests it may be useful to receive Agency feedback on the model risk assessment and prospective adequacy assessment.  A Credibility Assessment Plan is suggested for inclusion in pre-submissions.  For regulatory submissions, the Draft Guidance recommends inclusion of a Credibility Assessment Report.  The structure for both a Credibility Assessment Plan and Credibility Assessment Report are provided in Appendix 2 of the Draft Guidance. Id. at 34-36.

For regulatory and legal professionals, the Draft Guidance provides information that will help ensure regulatory submissions provide appropriate documentation to support the credibility of computational modeling and simulation information provided to support the safety and effectiveness of medical devices.  As FDA indicates, it is especially important to consider these issues within the clinical context (conditions of use).  A model that is credible in one context may not be credible in another.  We are interested to hear from engineers as to whether this guidance will also prove helpful in developing and validating CM&S.