Disclaimer: This article reflects the views of the authors and should not be construed to represent FDA’s views or policies.
In line with ISPE’s 2024–2029 Strategic Plan, which calls on members to engage with emerging and innovative topics and to lead novel integrations of epidemiology and public health into regulatory science, ISPE submitted feedback on three recent draft guidances from the U.S. Food and Drug Administration (FDA). These guidances, focusing on the use of artificial intelligence (AI), real-world data (RWD), and real-world evidence (RWE) to support regulatory decision-making, represent critical inflection points for our field.
ISPE’s review and comments to these guidances reflect its mission to “advance the health of the public” by supporting international policy, enhancing scientific rigor, and fostering collaboration across sectors and disciplines. Key to this stated goal is advocacy for patients, for caregivers, and for all stakeholders in the healthcare ecosystem. The coordinated ISPE response followed a structured process; ISPE members expressed interest following a call for volunteers and the core response group (strategic lead, co members) were selected based on the topic scientific expertise. The draft response was reviewed by the wider ISPE membership and final ISPE response was verified by the Board. This effort exemplifies how ISPE is actively promoting evidence-based healthcare policy while fulfilling its mission to advance the public’s health through methodologically sound, collaborative, and globally informed science.
Given the rapidly evolving nature of AI, this guidance outlines a framework for establishing the credibility of AI models to support regulatory decision-making regarding safety, effectiveness, or quality for drugs, including a seven-step risk-based assessment. ISPE’s comments focused on aligning AI/ML terminology with FDA’s digital health definitions, promoting model transparency, and encouraging practical application of the framework in real-world settings, especially in pharmacovigilance and drug safety. The feedback highlighted the importance of defining model risk in the context of influence and consequence, maintaining lifecycle performance, and improving clarity around use cases pertinent to RWE.
ISPE’s feedback emphasized several key recommendations to strengthen the draft guidance. We encouraged the FDA to clarify its terminology by aligning definitions of AI and machine learning (ML) with those in the Agency’s existing Digital Health and Artificial Intelligence Glossary and to clarify the scope of AI models referenced, including whether it encompasses generative AI and foundation models such as GPT-4. To ground the risk-based framework in practical application, we proposed real-world use cases—such as AI-driven pharmacovigilance systems that analyze electronic health records to detect adverse drug reactions—highlighting how the framework could be applied in post-market safety surveillance. We also stressed the importance of lifecycle management, especially for self-evolving systems, and recommended the establishment of a centralized, public-facing repository of validated AI models, similar to the EU AI Act’s registry for high-risk models, to enhance transparency and reusability. Lastly, we called for greater attention to model provenance and governance, encouraging the FDA to consider frameworks from organizations like the National Telecommunications and Information Administration (NTIA) and to set clear expectations around documentation, versioning, and traceability that are proportional to the model’s risk and regulatory impact.
These recommendations are consistent with recent perspectives shared by FDA leadership. In an October 2024 JAMA article, former FDA Commissioner Robert Califf and colleagues emphasized the need for AI regulation that is both flexible and rigorous, acknowledging that oversight of AI in drug development must include real-world performance evaluation and continuous monitoring. The article also highlighted the challenge of balancing innovation with risk mitigation and underscored that AI models must ultimately demonstrate benefit to patient outcomes, not just optimize internal efficiencies
The proposed FDA guidance focuses on the biological differences that can impact outcomes in clinical and non-interventional studies. As such, the guidance focuses on sex as assigned at birth, although ISPE recognizes the importance of – and advocates for – additional background information to provide a wider contextual perspective on why sex matters in clinical research. As part and parcel of this, several key points were raised in our ISPE response, of which the distinction between sex and gender – the latter defined not by biological sex but by cultural roles, behaviours, attributes, etc. – must be clarified at the outset of any such conversation. This is especially key when discussing data standards. While ISPE applauds the inclusion of gender data if evidence suggests that it plays a relevant role in trial outcomes, given that gender is a social construct subject to cultural variations, care must be taken to outline any such data collection with context. This is especially important in for clinical endpoints that may be recorded differently when sex (at birth) is considered versus gender (e.g., pain scores, which may be less likely to be recorded or recognized in women compared with men).
Crucial to accurately addressing sex differences in clinical studies is the matter of representation, which ISPE highlights in their response letter. In early-phase trials, sex-related information may be systematically limited and thus caution is needed that limited or lack of data on sex differences should not be translated as the absence of sex differences. Further, a historic inattention paid to female data in specific diseases (e.g. cardiovascular and Parkinson’s disease) due to lack of understanding of biological differences that may impact disease symptomatology and progression, access barriers, physician bias can adversely impact trial design and interpretation of study findings. When evidence on sex differences exists, attention is also needed on its quality, bias assessments and its impact on data standards with estimating power calculations. While ISPE supports the FDA guidance in this space, it also recommends widening the scope of the guidance, and recommended expanded strategies for recruitment and enrollment as discussed in the Clinical Trials Transformation Initiative and from recent developed evidence maps of related strategies. These additional resources will serve only to enhance the FDA’s efforts.
Furthermore, specific statistical concepts are relevant to fully address sex differences in the evaluation of medicinal products by the Agency; ISPE strongly recommends the expansion of baseline (i.e., pre-trial) data to support hypothesis-driven analyses powered to conduct sex-based analysis in clinical trials. Unless sponsors explicitly assess the potential link between sex and trial outcomes, the generalizability of findings will be compromised. For that reason, the collection of information should consider variables that may be important in understanding sex differences beyond the standard demographic characteristics, such as family and social factors. Finally, as event rates and risk factors can differ between sexes, studies must be powered to detect a wider breadth of confidence intervals for effect size, in order to properly represent both sexes in clinical trials.
While comprehensive in its breadth, the guidance nevertheless could provide additional insights by highlighting issues, such as the underrepresentation of women in the training, testing and validation of AI models, which can ultimately introduce bias and omission into AI-generated outputs. As with any data source, biases – whether intentional or unintentional – lead to inaccuracies and further threaten to address the needs of marginalized trial participants and to adversely affect downstream users of clinical data. ISPE applauds the FDA’s commitment to advancing regulatory science in this evolving space and looks forward to continued dialogue and opportunities to contribute to this important effort.
In September 2024, the Center for Devises and Radiological Health (CDRH) issued draft guidance to provide recommendations on how PPI can be collected and shared with the FDA. The draft guidance proposes updates to the current PPI guidance (which was finalized in 2016) in response (in part) to public comments submitted to FDA in June 2023. The proposed changes to the guidance would reflect the current application of PPI across the total product life cycle, the role of qualitative and quantitative preference methods for eliciting PPI, and how to conduct and submit PPI studies to the agency.
ISPE’s comments also focused on clarifying the contexts in which PPI can be useful in applications along the total product lifecycle. While the proposed guidance includes case studies, the potential application of PPI is broad and the agency’s views on preference-sensitive questions in applications beyond benefit risk assessment would support sponsors’ evidence planning.
The proposed changes to the guidance emphasize the potential role of formative qualitative research to inform the design of PPI studies. There are fewer preference-study specific guidelines for the conduct of qualitative preference research and fewer precedents for the use of qualitative PPI in FDA decision making. The current PPI guidance describes the recommended qualities of preference studies (most of which relate to quantitative studies), and it would be useful to guide sponsors in the design and execution of qualitative preference research that would yield results that the agency would consider scientific evidence for decision making.
The proposed guidance recommends consideration of the preferences of care partners and health care providers when they are relevant in decision making. The case for considering these stakeholders’ preferences is well-established when patients are cognitively impaired and/or pediatric. And understanding health care providers’ preferences is common when the focus is shared decision making. It would be useful to have clarification of the contexts and applications along the TPL in which preferences of cohorts other than (or in addition to) patients would be considered by FDA.
Finally, ISPE applauds FDA’s recognition of good practices in the health preference research literature. Where the literature is broad, ISPE suggests that FDA provide additional detail on its methodological guidance, including the types of preference heterogeneity analysis and the selection and interpretation of assessments of response consistency. Preference studies can be an important part of the evidence base and ISPE is pleased to support the FDA’s commitment to the application of good quality PPI study results to support patient-focused drug development throughout the total product lifecycle. Additional clarity regarding the agency’s recommendations may ensure that sponsors’ investments in this research is valuable for decision making.
Advancing Regulatory Science through Professional Collaboration
ISPE’s contributions in reviewing and commenting on these regulatory guidance documents reflect the Society’s ongoing commitment to promoting scientifically robust, transparent, and patient-centered approaches to evidence generation. The recent feedback efforts exemplify how ISPE members can actively participate in providing feedback to stakeholders to shape policy.
Looking ahead, ISPE remains committed to monitoring and responding to emerging regulatory guidances and encourages members to submit additional topics for consideration the ISPE Emerging/Advocacy Issue Checklists.
