Change is Inevitable – Plan Ahead: An Assessment of FDA’s Draft Guidance on Predetermined Change Control Plans for Artificial Intelligence/Machine Learning-Enabled Device Software Functions

FDA recently published a long-awaited draft guidance aimed at reducing the need for prior FDA authorization of modifications to artificial intelligence/machine learning (AI/ML)-enabled device software functions (ML-DSFs).  The draft guidance, “Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence/Machine Learning (AI/ML)-Enabled Device Software Functions,” describes the information on planned modifications that should be included with an initial premarket submission to enable FDA to prospectively evaluate a manufacturer’s process for implementing software changes.

History of the Development of Predetermined Change Control Plans

Traditionally, manufacturers seeking to make improvements or updates to their software device functions have needed to consider whether a new marketing submission is required, based on criteria specified in FDA regulations that focus on the potential impact of the proposed modifications on safety and effectiveness.  See 21 CFR 807.81(a)(3) and 21 CFR 814.39(a), and related guidance documents (e.g., here for a 510(k) device and here for a PMA device).  The purpose of a predetermined change control plan (PCCP) is to avoid the need for such post-market assessments – and the delay and expense entailed when it is determined that a new submission is needed – through prospectively defining a process for making post-market changes.

The draft guidance follows the December 2022 enactment of the Food and Drug Omnibus Reform Act of 2022 (FDORA).  FDORA added section 515C, “Predetermined Change Control Plans for Devices” to the Federal Food, Drug, and Cosmetic (FD&C) Act.  Section 515C authorizes FDA to approve a PCCP submitted as part of a PMA or 510(k) submission and to require that such PCCP include (a) labeling required for safe and effective use of the device as it changes; (b) notification requirements if the device does not function as intended pursuant to the plan, and (c) performance requirements for changes made under the plan.  Once FDA has cleared or approved a PCCP, that statute states that a supplemental application “shall not be required” to make a change consistent with such PCCP.

Although FDORA gave FDA express authority to allow PCCP , FDA had already been considering creating a mechanism for manufacturers to prespecify modifications for AI/ML-based software subject to 510(k) premarket notification and de novo classification for several years.

The 2019 discussion paper, “Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) – Discussion Paper and Request for Feedback,” introduced the PCCP principle as a means to “enable responsible performance enhancements in AI/ML technologies.”  The discussion paper described a PCCP as including (1) the types of anticipated modifications (SaMD Pre-Specifications or SPS) based on the retraining and model update strategy and (2) the associated methodology  (Algorithm Change Protocol or ACP) being used to implement those changes in a controlled manner to manage patient risks.

In February 2020, FDA granted a de novo classification request (DEN190040) for software intended to assist medical professionals in the acquisition of cardiac ultrasound images and that included a PCCP for future software modifications.  The reclassification order established a Special Control requiring the inclusion of a “detailed protocol that describes, in the event of a future, change, the level of change in the device technical specifications or indications for use at which the change or changes could significantly affect the safety or effectiveness of the device and the risks posed by these changes.”  Additionally, it stated that the “assessment metrics, acceptance criteria, and analytical methods used for the performance testing of changes that are within the scope of the protocol must be included.”

In January 2021, as part of its “Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) Action Plan,” FDA committed to issuing a draft guidance on PCCPs that would address the content of SPSs and ACPs and identify the types of modifications appropriate under a PCCP.  That draft guidance has now been issued.

Overview of the Draft Guidance

The draft guidance describes a PCCP as a “plan that includes device modifications that would otherwise require a premarket approval supplement, De Novo submission, or a new premarket notification.”  It states that FDA will review a PCCP for a ML-DSF as part of the initial marketing submission and “authorize” it in whole or in part, i.e., permit either all of the proposed modifications or only some of them.

Proposed modifications should be consistent with the device’s intended use and indications for use and be “intended to maintain or improve the safety and effectiveness of the device.”  Manufacturers should be able to verify and validate the proposed modifications within their existing quality system.

Post-market, manufacturers can make modifications consistent with the PCCP and document the modification in accordance with their quality system, without the need for a new marketing submission.  Modifications that are not consistent with the authorized PCCP, however, must be evaluated in accordance with the traditional regulatory process described in applicable regulations and guidances referenced above.

The draft guidance provides examples of modifications that may be included in a PCCP, including:

• modifications related to quantitative measures of the ML-DSF performance specifications, including improvements to analytical and clinical performance resulting from re-training the ML model based on new data from the same type of input signal;
• modifications related to device inputs to the ML-DSF which may involve including new sources of same signal type; and
• modification related device’s use and performance for use within a specific subpopulation that was part of the originally indicated population.

Components of a PCCP

The draft guidance describes the sections that should be included as part of a PCCP and the specific information that should be included under each section.  The sections are: (1) description of modifications; (2) modification protocol; and (3) impact assessment.

Description of Modifications

The PCCP should include a detailed description of the changes to “device characteristics and performance” that are expected to result from proposed modifications.  FDA recommends that PCCPs specify only a “limited number” of modifications but does not provide a limit on the number of modifications that can be included.

Both automatic and manual modifications can be included in a PCCP, so manufacturers should indicate the proposed implementation strategy for each modification.  In addition, manufacturers should identify whether the modifications will apply to all devices in the field (referred to as global adaptations) or will be implemented based on a specific clinical site or individual patients” (referred to as local adaptations).  For local adaptations, information should be included addressing what “local factors or conditions warrant a local change.”

FDA recommends manufacturers submit each modification, associated labeling changes and specific performance evaluation in the Modification Protocol (MP).

Modification Protocol

The modification protocol (MP) describes the methods that will be followed when “developing, validating, and implementing those modifications, to ensure the device remains safe and effective.” For each proposed modification, the MP should describe:

1. Data management practices: The MP should address how new data will be “collected, annotated, curated, stored, retained, controlled and used”; identify any relationship between the new data and the original data used for training and testing the initial device; define any data sequestration strategies and explain how the data are separated into training and testing dataset; and ensure that access to the testing datasets is restricted to prevent the use of that data for training.
2. Re–training practices: The MP should state whether the model will need to be re-trained. If retraining applies to only certain parts of the ML-DSF, manufacturers will need to demonstrate that re-training does not affect other functions or software components.  The MP should also discuss how the manufacturerr will address risks associated with overfitting and model bias when re-training.
3. Performance evaluation protocols: The MP should include plans to verify and validate modifications. If the proposed performance evaluation methods are different from the original methods used, the manufacturer should describe the differences and provide a justification for the changes.  Just like any other verification or validation activity, the protocol should include pre-defined acceptance criteria and any statistical tests used.  Since the manufacturer will be implementing the change without FDA premarket review, this section of the MP will need to include a statement that if there is a failure in performance, the failure will be recorded and the modification will not be implemented.
4. Update procedures: The MP should describe how the manufacturer will update the software and any associated labeling that will be required for the modifications. Updates performed automatically will face a “higher bar” for demonstrating safety and effectiveness.  For automatic updates, the MP will need to describe how the software will be updated; how legacy users may be affected by the modification;  how users will be made aware of the modification; and any impact on performance that could be caused by the modification (e.g., whether clinically meaningful differences may be observed in patient results before and after the modification).  Further, the MP should address how version information will be presented to the user for the modified device.  In addition, manufacturers may need to establish a device monitoring plan that will address how they will monitor post market device performance of the modifications, including monitoring real-world device performance or justify why that device monitoring is not necessary.

The draft guidance includes an appendix with example elements that may be required in a MP depending on the size, scope, complexity, and risk associated with each proposed modification.

Impact Assessment 

A PCCP should include an impact assessment that evaluates the risks and benefits of the modifications and how any risks will be mitigated.  The impact assessment can be included with the MP or provided as a stand-alone document, and should include:

1. a comparison of the version of the device without modification to the version of the device with the modifications implemented;
2. risks and benefits of the modifications;
3. how the MP will continue to reasonably ensure safety and effectiveness;
4. how the implementation of one modification affects the implementation of another; and
5. the cumulative impact of all modifications.

Things to Consider When Deciding Whether to Include a PCCP

PCCPs provide a new mechanism for introducing certain significant changes to an ML-DSF.  The use of a PCCP is not required; a manufacturer can continue to implement significant device changes via supplemental submissions to FDA.  However, including a PCCP can speed up the process of making post-market software modifications and potentially reduce costs by avoiding additional marketing submissions.

Once authorized, a PCCP is considered a technological characteristic of the authorized device.  Consequently, details of the PCCP would be included in the 510(k) Summary, De Novo decision summary, or PMA Summary of Safety and Effectiveness.  Manufacturers seeking to modify a PCCP once it has been authorized will need to submit the modified PCCP as part of a marketing submission for FDA’s review.  The PCCP is applicable only to the device that was the subject of the market authorization in which it was included, which means that, if the manufacturer makes other changes to the device before implementing the modifications outlined in the PCCP, a new submission for the modified device will be needed to reestablish the PCCP.

The Agency recommends that manufacturers discuss the PCCP strategy with FDA through the Q-Submission program before submitting a marketing submission with a PCCP.  We note that FDA has already updated the eStar template to include a separate section indicating whether the sponsor is including a PCCP with the submission and a mechanism to attach it to the submission.

Industry stakeholders seeking to provide feedback to FDA can submit comments to  Docket: FDA-2022-D-2628 by July 3, 2023.