Background

A machine learning-enabled device software function (ML-DSF) uses a type of artificial intelligence called machine learning to analyze data and make predictions or decisions based on that analysis. Examples of ML applications in medicine include earlier disease detection and diagnosis, development of personalized diagnostics and therapeutics, and development of assistive functions to improve the use of devices with the goal of enhancing user and patient experience.

Modifications to an ML-DSF that could affect the safety or effectiveness of the device require premarket authorization from FDA. However, in order to support the iterative development of ML-DSFs, FDA previously described how a “Predetermined Change Control Plan” (PCCP) could be included in a premarket submission for a device that is [or includes] an ML-DSF. By submitting a PCCP, a sponsor may obtain pre-approval for intended modifications (and their method of implementation) to an ML-DSF without necessitating additional marketing submissions for each modification delineated and implemented in accordance with the PCCP. FDA considers the PCCP to be part of the technological characteristics of the authorized device and stresses that while the PCCP can be part of a broader marketing submission for a device, it must be clearly delineated as a standalone section.

FDA’s April 2019 “Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD)” discussion paper and its January 2021 “Artificial Intelligence/Machine Learning (AI/ML) Software as a Medical Device Action Plan” (previously summarized online here)” proposed a framework for PCCPs, while promising that FDA would issue draft guidance at some point in the future.

Scope of the new draft guidance

The long-promised draft guidance on Predetermined Change Control Plans arrived on March 30, and it recommends that a PCCP include:

1. a detailed description of the specific, planned device modifications;

2. the associated methodology to develop, validate, and implement those modifications in a manner that ensures the continued safety and effectiveness of the device across relevant patient populations, referred to as the “Modification Protocol”; and

3. an Impact Assessment to describe the assessment of the benefits and risks of the planned modifications and associated risk mitigations.

The draft guidance describes how premarket authorization for an ML-DSF with a PCCP must be established through the 510(k), De Novo, or PMA pathway, as appropriate, as a PCCP must be reviewed and established as part of a marketing authorization for a device prior to a manufacturer’s implementing any modifications under that PCCP. For 510(k) submissions, in making a determination of substantial equivalence where the predicate device was authorized with a PCCP, the subject device must be compared to the version of the predicate device cleared or approved prior to changes made under the PCCP. The guidance notes that where a new PCCP is proposed for a previously authorized device with a PCCP, FDA “intends to focus its review on the aspects of the device that are most significantly modified.”

In announcing the draft guidance, FDA touted its PCCP approach as “the least burdensome” way to safely allow companies to modify products that use ML.

Description of Modifications

The draft guidance explains that the description of each planned modification to an ML-DSF included in the “Description of Modifications” section of a PCCP premarket authorization request should:

• identify the specific, planned modifications to the ML-DSF that the manufacturer intends to implement;

• enumerate the list of individual proposed device modifications discussed in the PCCP, as well as the specific rationale for the change to each part of the ML-DSF that is planned to be modified; and,

• describe modifications that are specific, and able to be verified and validated within the existing quality system of the device.

While not an exhaustive list, FDA outlines types of modifications which could be considered acceptable for authorization within a PCCP:

1. modifications related to quantitative measures of ML-DSF performance specifications;

2. modifications related to device inputs to the ML-DSF;

3. limited modifications related to the device’s use and performance (e.g., for use within a specific subpopulation).

FDA also stresses that, at this time, changes which affect the device’s intended use or indications for use are not considered appropriate for a PCCP. Examples of changes which may and may not be deemed acceptable under particular PCCPs are included in Appendix B of the draft guidance.

Modification Protocol

FDA advises in the draft guidance that a PCCP’s “Modification Protocol” section in a premarket submission should include the documentation describing the methods that will be followed when developing, validating, and implementing modifications delineated in the “Description of Modifications” section of the PCCP. The draft guidance identifies four primary components of a Modification Protocol that should summarize a manufacturer’s:

1. Data management practices, which outline how new data will be collected, annotated, curated, stored, retained, controlled, and used by the manufacturer for each modification.

2. Re-training practices, which are the processing steps that are subject to change for each modification and the methods that will be used by the manufacturer to implement modifications to the ML-DSF.

3. Performance evaluation protocols, which describe the processes that will be followed to validate that the modified ML-DSF will meet the specifications identified as part of a specific modification, in addition to maintaining the specifications that are not part of the modification but may be impacted by it.

4. Update procedures, which describe how manufacturers will update their devices to implement the modifications, provide appropriate transparency to users, and, if appropriate, updated user training about the modifications and perform real-world monitoring, including notification requirements if the device does not function as intended pursuant to the authorized PCCP.

FDA recommends that each of the four primary components of a Modification Protocol be addressed for each planned modification covered in the Description of Modifications. Example elements of each of the four Modification Protocol components are provided in Appendix A of the draft guidance.

Upon FDA review of a PCCP, it is possible that FDA may determine that a Modification Protocol supports some but not all modifications identified in a PCCP; in such cases, only those modifications that are appropriate in FDA’s findings of substantial equivalence or reasonable assurance of safety and effectiveness would be included in the authorized PCCP.

Impact Assessment

An “Impact Assessment,” in the context of a PCCP, is defined as “the documentation of the assessment of the benefits and risks of implementing a PCCP for an ML-DSF, as well as the mitigations of those risks.” The draft guidance says the manufacturer’s existing quality system should be used as the framework in which to conduct an Impact Assessment for the modifications set forth in the PCCP. Documentation for an Impact Assessment provided to FDA in a marketing submission containing a PCCP should:

1. compare the version of the device with each modification implemented to the version of the device without any modifications implemented;

2. discuss the benefits and risks, including risks of social harm, of each individual modification;

3. discuss how the activities proposed within the Modification Protocol continue to reasonably ensure the safety and effectiveness of the device;

4. discuss how the implementation of one modification impacts the implementation of another; and,

5. describe the cumulative impact of implementing all modifications.

Diversity data

Pursuant to FDA’s stated prioritization of advancing health equity, the draft guidance also says medical device sponsors should clearly communicate to the agency performance considerations with respect to race, ethnicity, disease severity, gender, age, and geographical considerations, as part of the ongoing development, validation, implementation, and monitoring of AI/ML-enabled devices. The guidance further recommends “that manufacturers consider additional characteristics, such as those described in the Blueprint for an AI Bill of Rights.

Next steps

We expect the guidance, once finalized, to encourage innovation and delivery of AI/ML-enabled medical devices by enabling manufacturers to make certain updates to their devices without re-engaging FDA prior to their implementation. However, it also puts the onus on medical device manufacturers to plan ahead and approach FDA through the Q-submission process to discuss a proposed PCCP, and then include a lengthy change control section in their marketing submissions in order to obtain pre-authorization for such updates.

FDA has invited comments on the draft guidance through July 3, 2023.