Nearly three years have passed since the US FDA, Health Canada, and UK MHRA jointly identified ten guiding principles in June 2024 in which built upon existing practices for machine learning-enabled medical devices (MLMDs). These principles were developed by the group following industry consultation.

In this article, we provide an overview of the ten guiding principles for Good Machine Learning Practice (GMLP) in Machine Learning-enabled Medical Devices (MLMD) development under the FDA, as well as the five guiding principles for Predetermined Change Control Plans (PCCPs), in light of additional FDA regulations concerning AI systems.

AI/Machine Learning Devices: Best Practices for GMLP for MLMD Development

The AI/ML medical device field is one of the fastest developing areas in the medical device industry, with a significant increase of 510(k) and De Novo submissions to the FDA for such devices in the last three years. While most regulatory agencies are racing to catch up with the rate of industrial innovation in this space, the FDA has made significant strides in publishing relevant guidance on this topic to facilitate reviews of regulatory submissions as part of its AI/ML Software as a Medical Device (SaMD) Action Plan published by the FDA in 2021. GMLP is a set of best practices for AI/ML in areas such as training, evaluation and documentation, data management, and interpretations, similar to those best practices for quality management systems or software engineering. 

Leveraging experience on AI/ML technologies utilized in other industries, primarily as it relates to cybersecurity initiatives developed by the US Cybersecurity & Infrastructure Security Agency (CISA), the aim of the guiding principles include: 

• • The adoption of good practices that have been demonstrated in other sectors 
  • Revise practices from other sectors so that they are relevant and applicable to medical technology and the healthcare sector 
  • Develop new practices specific to medical technology and the healthcare sector 

The following provides an overview of the ten guiding principles for GMLP for MLMD development: 

Ten Guiding Principles for GMLP for MLMD Development

1.  Multi-Disciplinary Expertise Is Leveraged Throughout the Total Product Life Cycle:

   In-depth understanding of a model’s intended integration into clinical workflow, and the desired benefits and associated patient risks, can help ensure that ML-enabled medical devices are safe and effective and address clinically meaningful needs over the lifecycle of the device.

2. Good Software Engineering and Security Practices Are Implemented:

   Model design is implemented with attention to the “fundamentals”: good software engineering practices, data quality assurance, data management, and robust cybersecurity practices. These practices include methodical risk management and design process that can appropriately capture and communicate design, implementation, and risk management decisions and rationale, as well as ensure data authenticity and integrity.

3. Clinical Study Participants and Data Sets Are Representative of the Intended Patient Population:

   Data collection protocols should ensure that the relevant characteristics of the intended patient population (for example, in terms of age, gender, sex, race, and ethnicity), use, and measurement inputs are sufficiently represented in a sample of adequate size in the clinical study and training and test datasets, so that results can be reasonably generalized to the population of interest. This is important to manage any bias, promote appropriate and generalizable performance across the intended patient population, assess usability, and identify circumstances where the model may underperform.

4. Training Data Sets Are Independent of Test Sets:

   Training and test datasets are selected and maintained to be appropriately independent of one another. All potential sources of dependence, including patient, data acquisition, and site factors, are considered and addressed to assure independence.

5. Selected Reference Datasets Are Based Upon Best Available Methods:

   Accepted, best available methods for developing a reference dataset (that is, a reference standard) ensure that clinically relevant and well characterized data are collected, and the limitations of the reference are understood. If available, accepted reference datasets in model development and testing that promote and demonstrate model robustness and generalizability across the intended patient population are used.

6. Model Design Is Tailored to the Available Data and Reflects the Intended Use of the Device:

   Model design is suited to the available data and supports the active mitigation of known risks, like overfitting, performance degradation, and security risks. The clinical benefits and risks related to the product are well understood, used to derive clinically meaningful performance goals for testing, and support that the product can safely and effectively achieve its intended use. Considerations include the impact of both global and local performance and uncertainty/variability in the device inputs, outputs, intended patient populations, and clinical use conditions.

7. Focus Is Placed on the Performance of the Human-AI Team:

   Model design is implemented with attention to the “fundamentals”: good software engineering practices, data quality assurance, data management, and robust cybersecurity practices. These practices include methodical risk management and design process that can appropriately capture and communicate design, implementation, and risk management decisions and rationale, as well as ensure data authenticity and integrity.

8. Testing Demonstrates Device Performance During Clinically Relevant Conditions:

   Statistically sound test plans are developed and executed to generate clinically relevant device performance information independently of the training data set. Considerations include the intended patient population, important subgroups, clinical environment and use by the Human-AI team, measurement inputs, and potential confounding factors.

9. Users Are Provided Clear, Essential Information:

   Users are provided ready access to clear, contextually relevant information that is appropriate for the intended audience (such as health care providers or patients) including: the product’s intended use and indications for use, performance of the model for appropriate subgroups, characteristics of the data used to train and test the model, acceptable inputs, known limitations, user interface interpretation, and clinical workflow integration of the model. Users are also made aware of device modifications and updates from real-world performance monitoring, the basis for decision-making when available, and a means to communicate product concerns to the developer.

10. Deployed Models Are Monitored for Performance and Re-training Risks Are Managed:

    Users are provided ready access to clear, contextually relevant information that is appropriate for the intended audience (such as health care providers or patients) including: the product’s intended use and indications for use, performance of the model for appropriate subgroups, characteristics of the data used to train and test the model, acceptable inputs, known limitations, user interface interpretation, and clinical workflow integration of the model. Users are also made aware of device modifications and updates from real-world performance monitoring, the basis for decision-making when available, and a means to communicate product concerns to the developer.

Guiding Principles for PCCPs for Machine Learning-Enabled Medical Devices (MLMDs)

The FDA has integrated the use of Predetermined Change Control Plans PCCPs into its AI/ML regulatory framework as a way of managing certain device changes where regulatory review/authorization is necessary before such changes can be implemented in marketed devices. This change management process helps to support the safety and effectiveness of the device throughout its total lifecycle. 

PCCPs describe plans proposed by a manufacturer that specifies the following: 

• • Certain planned modifications to a device 
  • The protocol for implementing and controlling those modifications and 
  • The assessment of impacts from modifications 

The objectives of the five guiding principles for PCCPs for MLMD established in 2021 were to: 

• • Provide foundational considerations that highlight the characteristics of robust PCCPs 
  • Facilitate and foster continual engagement and collaboration among stakeholders on the PCCP concept for MLMD 
  • Lay a foundation for PCCPs and encourage international harmonization, helping to support the advancement of responsible innovations in digital medical technology 

The following provides an overview of the guiding principles for PCCPs for MLMDs: 

Five Guiding Principles for PCCPs for MLMDs:

1. Focused and Bounded:

   A PCCP describes specific changes that a manufacturer intends to implement. Such changes are limited to modifications within the intended use or intended purpose of the original MLMD. This characterization can include:

   • • The extent of planned changes and scope of the MLMD with changes implemented
     • Plans in place to safely modify the device within the bounds of the PCCP, including methods for verifying and validating the changes and mechanisms to detect and revert or stop implementation of a change that fails to meet specified performance criteria
     • The impacts of the planned changes
2. Risk-based:

   The value and reliability of a PCCP is strengthened when the intent, design, and implementation of a PCCP are driven by a risk-based approach that adheres to the principles of risk management.

   This risk-informed perspective is relevant:

   • • Throughout the TPLC, from inception, through implementation and to use
     • To ensure that individual and cumulative changes remain appropriate over time for the device and its use environment
3. Evidence-based:

   Data collection protocols should ensure that the relevant characteristics of the intended patient population (for example, in terms of age, gender, sex, race, and ethnicity), use, and measurement inputs are sufficiently represented in a sample of adequate size in the clinical study and training and test datasets, so that results can be reasonably generalized to the population of interest. This is important to manage any bias, promote appropriate and generalizable performance across the intended patient population, assess usability, and identify circumstances where the model may underperform.

4. Transparent:

   For PCCPs, the best practice is to provide clear and appropriate information and detailed plans for ongoing transparency to users and other stakeholders. This helps ensure that stakeholders stay aware of the device’s performance and use before and after changes are implemented. Consider, for example:

   • • Characterization of data used in development and modifications, demonstrated to reflect the intended population
     • Comprehensive testing for planned changes
     • Characterization of the device before and after implementation of changes
     • Monitoring, detection, and response to deviations in device performance

5. Total Product Lifecycle (TPLC) Perspective:

   Creating and using a PCCP from a TPLC perspective can:

   • • Elevate the quality and integrity of a PCCP by continually considering the perspectives of all stakeholders as well as risk management practices throughout the TPLC

• • • Use and support existing regulatory, quality, and risk management measures throughout the TPLC to ensure device safety by monitoring, reporting and responding to safety concerns

EU Regulation 2024/1689 Harmonized Rules on Artificial Intelligence  

In comparison to the US, the EU recently released regulations concerning the legal framework regarding the development, placing on the market, putting into service, and the use of artificial intelligence systems (AI systems) in the EU. The EU is consistent with US, Canada, and the UK regarding their thoughts on AI and believe that AI is a fast-evolving family of technologies that benefit a wide variety of industries and social activities. Depending on its application, however, AI may pose certain risks and cause harm to the public interest and certain fundamental rights protected by law, generating the need for a legal framework to regulate the health, safety, and fundamental rights concerning high-risk AI systems. The EU considers certain medical devices and in-vitro medical (IVD) devices that utilize AI technology as high-risk AI systems. Determine if your medical device or IVD is considered a high-risk AI system by referring to Annex III here. 

Learn More About Good Machine Learning Device Practices Regarding AI/ML with MedEnvoy

This article provides an overview of the guiding principles for GLMP for the development and PCCPs of MLMD. Should you have any questions regarding medical device requirements for AI/ML or require relevant consulting support, get in touch. 

Related resources: