FDA has recently published two much-anticipated items related to artificial intelligence (“AI”), one pertaining to medical devices and the other to drug manufacturing. FDA has cleared or approved hundreds of products that work with AI or are AI-enabled and acknowledges that both the agency and industry need help navigating how FDA-regulated products engage with AI. Further, these pieces provide insight into FDA’s current thinking on this technology. FDA issued the draft guidance, “Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence/Machine Learning (AI/ML)-Enabled Device Software Functions,” in April 2023, with comments to be submitted by July 3, 2023. FDA also published the discussion paper, “Artificial Intelligence in Drug Manufacturing,” in May 2023, following comments from the public.

In a statement by FDA following the latest drug manufacturing discussion paper, the agency notes that, “Artificial intelligence (AI) and machine learning (ML) are no longer futuristic concepts; they are now part of how we live and work. . . Ultimately, AI/ML can help bring safe, effective, and high-quality treatments to patients faster.” However, while FDA understands the benefits that AI can offer, it also understands that many challenges accompany AI.

We will offer a high-level overview of the items discussed in these pieces, as well as some insights for industry.

Overview of FDA’s Draft Guidance for Medical Devices

In this draft guidance, FDA highlights a major concern for AI/ML-enabled medical devices, predetermined change control plans (“PCCP”). FDA hopes the guidance will promote safe and effective devices that use ML models trained by ML algorithms. One primary benefit of an AI/ML-enabled medical device is the product’s ability to improve ML model performance through iterative modifications. To support this development, FDA provides recommendations on the marketing submission content for a PCCP. This applies to marketing submissions through the 510(k) pathway, the PMA pathway, or the De Novo pathway, but the guidance may also apply to components of a combination product.

FDA considers the PCCP to be part of the technological characteristics of the devices, and generally expects this PCCP to contain:

• a detailed description of the specific, planned device modifications;
• the associated methodology to develop, validate, and implement those modifications in a manner that ensures the continued safety and effectiveness of the device across relevant patient populations, referred to as the “Modification Protocol”; and
• an Impact Assessment to describe the assessment of the benefits and risks of the planned modifications and risk mitigations.

The PCCP should be a standalone section of the marketing submission.

• The PCCP should be discussed in the marketing submission as part of the device description, labeling, and relevant sections used for determining substantial equivalence or reasonable assurance of safety and effectiveness.
• The product labeling should explain to the buyer that the device incorporates ML and has a PCCP, so that the buyer understands that they may need to perform software updates, and that such software updates may modify the device’s performance, inputs, or use.

The PCCP is reviewed and established as part of the marketing submission and considered part of the marketing authorization.

• Specifically, FDA highlights that deviation from the PCCP could significantly affect the safety or effectiveness of the device, such that modifications made to the device that are not in accordance with the submitted PCCP would likely require a new marketing submission.

Certain modifications can be made to the machine learning device software functions (“ML-DSF”), if properly included in the Description of Modifications section of a PCCP.

• Namely, FDA urges applicants to submit the “Description of Modifications” section with specific, limited modifications that can be verified and validated, and outlines what types of modifications are acceptable. No such changes should affect safety or efficacy of the product, or its intended use.
  • This is a high bar for ML devices, but adds functionality based on the data set available to the medical device.
• FDA provides examples of what types of modifications would be acceptable.
• FDA can determine in its review of a PCCP that some, but not all, modifications included are appropriate. As such, FDA may include only those modifications that it finds to have substantial equivalence of reasonable assurance of safety and effectiveness in the authorized PCCP.
• FDA recognizes that ML-DSFs where modifications are implemented automatically have an additional degree of complexity and will use its past experience and a risk-benefit assessment to review these PCCPs appropriately.
  • It is unclear how exactly FDA plans to alter its approach to these more complex devices.

There must be a Modification Protocol section as well, for those modifications provided in the Description of Modification section.

• The PCCP sections should be traceable such that FDA can easily tell which Modification Protocol is applicable to with modification. FDA recommends a table approach for ease of understanding.
• Data management practices in the Modification Protocol Section should demonstrate the training and testing methods utilized to identify and eliminate bias in the data and improve the robustness and resilience of the algorithms to withstand changing clinical inputs and conditions.
  • FDA demonstrates in the guidance its concern for significant modifications beyond those specified in the PCCP. This recommendation may speak to how a manufacture can try to limit changes to the algorithm that expand to the point of being considered a modification outside of a PCCP, just by the nature of the ML.

The PCCP should also contain an Impact Assessment documenting the benefits and risks of implementing a PCCP, and how these risks can be mitigated.

FDA is clearly concerned with the potential impact and expected hiccups of PCCPs for AI-enabled medical device software functions. The agency is asking applicants to review the process from start to finish and demonstrate diligence as to the risk and procedure for enacting modifications.

One major obstacle is that machine learning, by its nature, is constantly improving based on new information. This begs the question of whether there is any amount of learning that would turn the normal course of operation into a modification. Manufacturers would likely prefer to list as broad of a use case as possible in hopes of capturing possible unknown future uses of data. However, the guidance substantially limits which modifications are acceptable without a new marketing submission. By the guidance, manufacturers are to submit limited, specific modifications. This limitation could lead to a major uptick in marketing submissions, as manufacturers try to keep up with modifications that fall outside of the modifications addressed in the authorized PCCP.

We wonder if FDA may consider a streamlined submission for such marketing submissions to avoid a substantive review for a large quantity of these submissions. If you have any comments before July 3, 2023, or want to view others’ comments, the docket is available here.

Overview of FDA’s Discussion Paper for Drug Manufacturers

In this FDA discussion paper, “Artificial Intelligence in Drug Manufacturing,” FDA acknowledges that facilitating innovation in drug manufacturing is squarely within the best interest of public policy. FDA describes the areas of consideration where FDA would like public feedback, but does not intend for this list to be exhaustive. These areas of consideration are:

1. Cloud application may affect oversight of pharmaceutical manufacturing data and records;
2. The Internet of Things (“IOT”) may increase the amount of data generated during pharmaceutical manufacturing, affecting existing data management practices;
3. Applicants may need clarity about whether and how the application of AI in pharmaceutical manufacturing is subject to regulatory oversight;
4. Applicants may need standards for developing and validating AI models used for process control and to support release testing; and
5. Continuously learning AI systems that adapt to real-time data may challenge regulatory assessment and oversight.

We will discuss the highlights of these considerations in turn.

Cloud application may affect oversight of pharmaceutical manufacturing data and records.

• FDA underscores the critical importance of cybersecurity and makes it clear that an entity cannot abdicate responsibility for data security by contracting with a third-party hosting provider. In order for an entity to meet its regulatory obligations, entities must perform initial due diligence and have processes in place to periodically confirm the level of data security controls that a vendor has in place.
• FDA acknowledges that, with proper manufacturer oversight, current good manufacturing practice (“CGMP”) functions are often provided through third parties. So, this due diligence should be conducted to identify gaps in quality agreements relating to managing risks associated with AI are addressed. Further, FDA must consider how use of AI by third parties may affect the agency’s inspection process.
• As AI becomes more prominent, companies should update their required data security terms to include safeguards addressing AI data security.

The IOT may increase the amount of data generated during pharmaceutical manufacturing, affecting existing data management practices.

• AI may generate more data about processes and products. In response, FDA is considering the need to revise its previous guidance on data retention and metadata to reflect a new balance of data integrity and retention with the logistics of data management for increased amounts of raw data.
• Currently, there is little to no guidance on how entities should leverage data sampling, data aggregation, and other data analytics tools to inform decisions on processes and products. FDA acknowledges that data analytics will be a key part of decisioning in the future and is looking for input as to the best practices for such analytics.

Applicants may need clarity about whether and how the application of AI in pharmaceutical manufacturing is subject to regulatory oversight.

• FDA acknowledges that with respect to technology in its many forms, distinctions regarding areas of FDA oversight have become increasingly blurred, such as CGMP compliance or applications for market approval.
• Regulatory uncertainty can stifle innovation and FDA is signaling that it seeks to avoid the uncertainty with a clear paradigm of what is covered by any FDA regulations and how they can produce guidance for industry to assist with this uncertainty.

Applicants may need standards for developing and validating AI models used for process control and to support release testing.

• FDA acknowledges that it’s not enough to regulate the use of AI models in isolation, there must be guidance around training and updating the algorithms that power the AI process. However, FDA notes the lack of guidance for development and validation of AI models that could affect product quality, and for how to factor in the potential to transfer learning from one AI model to another.
• Of particular interest is that FDA echoes the concerns of algorithmic bias in AI applications that other governmental entities worldwide have started examining.
  • For example, the European Union is considering potential legislation, the Artificial Intelligence Act, focusing on regulating the development and use of AI, which discusses these issues surrounding AI bias in algorithms.

Continuously learning AI systems that adapt to real-time data may challenge regulatory assessment and oversight.

• The last point FDA makes is a truly novel and interesting point. FDA leaves the door open that a day may come where AI informs regulatory decisions, rather than regulatory decisions driving the development of AI. Specifically, FDA notes that applicants may need clarity on:
  • the expectations for verification of model lifecycle strategy and the approach for FDA’s examination of continuously updated AI control models during a site inspection; and
  • expectations for establishing product comparability after changes to manufacturing conditions introduced by the AI model, especially for biological products.
• How does a cumbersome and slow-moving federal agency compete and stay timely in a world where data and information move and evolve so quickly?

FDA recognizes the changes happening at a quick pace and is trying to figure out how its regulations fit for these evolutions. The agency acknowledges that it does not have all of the answers and has opened the door for industry to offer its insight.