Artificial intelligence is no longer a future-state concept in pharmaceutical manufacturing. Machine learning models are being deployed today in batch record review, deviation prediction, chromatography data analysis, and visual inspection systems. These are GxP-impacting systems, and they need to be validated. The challenge is that traditional validation frameworks were designed for deterministic software, not probabilistic models whose outputs change as they encounter new data.
This article outlines a practical framework for validating AI and ML systems in GxP environments, drawing on emerging regulatory guidance from FDA, EMA, and MHRA, as well as hard-won lessons from pharmaceutical companies already navigating this space.
The Regulatory Landscape
Regulators have been signaling their position on AI/ML for several years. FDA's 2021 action plan for AI/ML-based Software as a Medical Device (SaMD) and subsequent discussion papers laid important groundwork. The agency distinguishes between "locked" algorithms, whose behavior is fixed at release, and "adaptive" algorithms, which continue to learn post-deployment. This distinction carries significant validation implications.
EMA's reflection paper on the use of artificial intelligence in the lifecycle of medicines (2023) takes a broader view, addressing AI used in drug development, manufacturing, and regulatory submissions. It emphasizes the need for transparency in model design, training data governance, and ongoing monitoring. MHRA's guidance on AI and machine learning similarly stresses algorithmic accountability and auditability.
What all three agencies have in common is an expectation that organizations can demonstrate they understand their AI systems, control the conditions under which they operate, and have processes in place to detect and respond when model performance drifts.
Regulators are not asking whether your AI works. They are asking whether you know when it stops working, and what you do about it.
Why Traditional Validation Falls Short
The validation frameworks most pharma organizations use today were built for software systems that behave predictably given the same inputs. You write a test script, execute it, compare output to expected values, and document the result. AI models do not behave this way. The same input can produce different outputs depending on the model version, the training data used, and the distribution of the production data the model has been exposed to since deployment.
Traditional IQ/OQ/PQ protocols can still apply to the infrastructure surrounding an AI system, but they cannot substitute for model-specific validation activities. Consider the following gaps:
- Performance characterization is probabilistic. You cannot verify a neural network the way you verify a lookup table. Acceptance criteria must be statistical, not binary.
- Test data governance is complex. The data used to evaluate a model must not overlap with the data used to train it, and both datasets must meet GxP data integrity requirements.
- Change control is continuous. A model retrained on new data is a changed system, even if the code is identical. Traditional change control processes were not designed for this frequency or type of change.
- Failure modes are different. Deterministic software either works or it does not. AI models degrade gracefully and silently, producing plausible but incorrect outputs that may go undetected for extended periods.
A Practical Framework for AI System Governance
Effective AI governance in GxP environments requires a purpose-built framework that sits alongside, not in place of, your existing validation infrastructure. The framework should address four domains: intended use and risk classification, model development governance, deployment and monitoring controls, and change and retirement management.
Intended Use and Risk Classification
Before any model development begins, the intended use must be precisely defined. This is not unique to AI, but it is more consequential here because the intended use boundary directly determines the scope of validation activities and the acceptable performance thresholds. A model that flags deviations for human review carries a different risk profile than one that automatically releases batches.
Risk classification should consider both the likelihood of model error and the severity of the consequence if an error reaches the patient. GxP systems directly impacting product quality, safety, or efficacy warrant the highest level of scrutiny. Systems used in administrative or non-GxP workflows may be handled under lighter-weight approaches consistent with CSA principles.
Model Development Governance
The model development phase is where most validation foundation is laid. This includes data governance, algorithmic design documentation, and performance benchmarking. Each of these activities must be controlled under your quality management system, with contemporaneous records maintained in a format that supports regulatory inspection.
Training data selection criteria must be documented and justified. The data itself must meet ALCOA+ principles: it must be attributable, legible, contemporaneous, original, accurate, complete, consistent, enduring, and available. For AI systems, this extends to the training pipeline, data preprocessing logic, and any data augmentation techniques applied.
Model Lifecycle Management and Validation
Model validation is not a one-time event. It is a lifecycle activity. At initial deployment, validation establishes that the model performs acceptably within its intended use envelope. But that envelope can shift as the real-world data distribution changes, as the process it monitors evolves, or as the organization's performance expectations increase.
A model validation plan should specify:
- Performance metrics and their acceptance thresholds (accuracy, precision, recall, F1, AUC-ROC, or domain-specific equivalents)
- The holdout or test dataset used for initial validation, with provenance and integrity controls
- Revalidation triggers - what events or conditions require a formal revalidation
- Ongoing monitoring metrics and their review frequency
- Escalation procedures when performance falls outside acceptable bounds
Cross-validation during development and prospective validation at deployment are both necessary but serve different purposes. Development-phase cross-validation answers whether the model generalizes from training data. Prospective validation answers whether it generalizes to the actual production environment under real operating conditions.
Data Integrity Considerations for AI Training Data
Data integrity has always been central to GxP compliance, but AI introduces new dimensions that existing guidance does not fully address. When a model is trained on historical batch records, laboratory results, or process sensor data, the quality of that data directly determines the quality of the model. Garbage in, garbage out is not a cliché in this context - it is a regulatory risk.
Several specific data integrity concerns arise in AI system contexts:
- Data lineage. You must be able to trace every training example back to its source system and demonstrate that it was generated, transferred, and stored in accordance with applicable data integrity controls.
- Data versioning. Training datasets must be version-controlled. If a model is retrained, you must be able to reconstruct the exact dataset used for any given model version.
- Labeling integrity. For supervised learning systems, the labels attached to training examples are as important as the examples themselves. Labeling processes must be controlled and documented, with qualification requirements for human labelers.
- Bias and representativeness. Training data that over-represents certain operating conditions or under-represents rare but critical events can produce models that perform well on average but fail in the scenarios that matter most.
Transparency, Explainability, and Auditability
Regulatory expectations for explainability are increasing. FDA's AI action plan and EMA's reflection paper both reference the need for organizations to be able to explain why a model produced a particular output. This does not necessarily mean that every model must be a simple decision tree, but it does mean that the organization must have a principled approach to interpretability and must be able to demonstrate that approach to inspectors.
At a minimum, the validation documentation should capture the model architecture and rationale for selecting it, the features used as inputs and their clinical or operational significance, the performance characteristics across different subpopulations or operating conditions, and known limitations and the boundaries within which the model is expected to perform reliably.
For high-risk applications, post-hoc explainability tools such as SHAP (SHapley Additive exPlanations) or LIME (Local Interpretable Model-agnostic Explanations) can provide case-level explanations that support audit trails. These outputs should be logged alongside model predictions in the audit trail so that any prediction can be reconstructed and explained at the time of inspection.
Auditability extends beyond explanations. The complete model provenance, including training data, hyperparameters, training run metadata, and evaluation results, should be stored in a controlled repository with access controls and version history consistent with 21 CFR Part 11 requirements where electronic records are involved.
Change Management for Continuously Learning Models
Change management is where most AI governance frameworks encounter the greatest friction with existing quality systems. Traditional change control processes assume that changes are discrete, infrequent, and human-initiated. Continuously learning models challenge all three assumptions.
Organizations need to make a deliberate architectural decision: will the production model update automatically as it processes new data, or will updates be gated through a formal change control and revalidation process? Both approaches are defensible, but the former requires substantially more sophisticated monitoring infrastructure and faster escalation pathways.
For most GxP applications, a gated update model is preferable. Under this approach, the model in production is frozen. New data is accumulated and used to develop a candidate replacement model. That candidate is validated against pre-specified acceptance criteria. Only upon successful validation is the candidate promoted to production, with a full change record documenting the rationale, the validation evidence, and the impact assessment.
The frequency of this cycle should be specified in the model governance plan and reviewed annually. Annual retraining may be appropriate for stable processes, while more dynamic environments may warrant quarterly model reviews.
Regulatory Expectations in Practice
FDA, EMA, and MHRA are aligned on several core expectations that organizations should translate directly into their AI governance procedures.
FDA expects organizations to demonstrate that AI/ML systems support, rather than replace, the scientific and quality judgment of qualified personnel. The agency's emphasis on the "predetermined change control plan" for adaptive algorithms signals that it wants to see the change management lifecycle defined and approved before the system is deployed, not retrofitted after the first model drift is detected.
EMA takes a lifecycle view, expecting that AI used in regulatory submissions and manufacturing is supported by documentation covering the full development and validation history. The agency has been explicit that model opacity is not acceptable for high-risk applications, and that sponsors must be able to provide meaningful explanations of model behavior during scientific advice and regulatory review meetings.
MHRA aligns with EMA's position on transparency and has additionally emphasized the importance of human oversight mechanisms. For any AI system with GxP impact, there should be a named human responsible for monitoring model performance and for making the decision to retrain, suspend, or retire the model.
Getting Started
For organizations building their first AI governance framework, the temptation is to wait for fully harmonized regulatory guidance before committing to a position. This is the wrong approach. Regulators expect to see organizations applying existing GxP principles thoughtfully and consistently to novel technologies, not waiting for prescriptive rules that may take years to arrive.
Start with a fit-for-purpose AI system inventory. Classify each system by GxP impact and model type. Identify where you have gaps in your existing procedures - data integrity controls for training data, change control for model updates, and audit trail requirements for model outputs are the most common. Build governance procedures that address those gaps, train your quality and IT teams, and apply the framework to your highest-risk systems first.
The validation principles that have served pharma for decades - document what you do, do what you document, and demonstrate that it works - apply equally to AI systems. The methods are different, but the discipline is the same.
Back to Insights