Generative AI Risk Factors: A Comprehensive Assessment Framework

Generative AI & Agentic AI Risk Factors

Data Source and Storage Attributes

Protected Health Information (PHI/ePHI) Usage

Risk Rating: 5 (High Risk)

Using protected health information in AI systems presents the highest level of risk. PHI includes individually identifiable health information such as patient medical records, lab results, diagnoses, prescriptions, and insurance details.  The high risk rating is justified because:

• PHI breaches can result in severe HIPAA violations with significant financial penalties

• Unauthorized access to PHI can lead to identity theft, insurance fraud, and patient harm

• Re-identification risks remain even when data is supposed to be de-identified, as demonstrated by studies showing algorithms can re-identify up to 85.6% of adults in certain datasets despite removal of protected health information identifiers

• Healthcare organizations have legal and ethical obligations to protect this data at the highest level

Personally Identifiable Information (PII) Usage

Risk Rating: 4 (Moderately High Risk)

PII usage in healthcare AI presents substantial but slightly lower risk than PHI. PII encompasses data capable of uniquely identifying an individual, such as name, address, phone number, and medical record numbers. The risk rating reflects:

• Violations still incur significant regulatory penalties

• Healthcare providers are legally obligated to protect PII under regulations such as HIPAA

• Identity theft risks are substantial

• The difference from PHI is that PII may not always reveal sensitive health conditions

Sensitive Internal Data (Financial/Operational)

Risk Rating: 3 (Moderate Risk)

Using sensitive internal data like financial information, strategic planning, or business operations data in AI systems presents moderate risk:

• While not directly affecting patient privacy, breaches can damage institutional reputation

• May expose competitive information or strategic plans

• Financial data breaches can lead to fraud or market manipulation

• Regulatory impact is typically less severe than with patient data unless involving billing fraud

Public Data

Risk Rating: 2 (Low-Moderate Risk)

Using publicly available datasets for AI training and operations presents lower but still notable risk:

• Even with public data, AI systems can sometimes identify patterns that could be linked back to individuals when combined with other datasets

• Quality and bias concerns in public datasets can lead to faulty AI outputs

• Less regulatory exposure since data is already public

• Still requires proper documentation of data provenance and usage permissions

API and Integration Attributes

Public API Usage

Risk Rating: 4 (Moderately High Risk)

Integrating with public APIs presents significant risk in healthcare AI implementations:

• Public APIs create external attack vectors into healthcare systems, potentially exposing sensitive data

• Healthcare organizations using public APIs must navigate complex regulatory landscapes that weren’t designed for AI integration

• Data transmitted through public APIs may be vulnerable during transit

• Public APIs may have their own security vulnerabilities and reliability issues

• Third-party dependencies create additional compliance challenges

Internal API Usage

Risk Rating: 3 (Moderate Risk)

Internal APIs present moderate risk when used in healthcare AI implementations:

• More controlled than public APIs but still create potential vulnerabilities

• Internal APIs may connect systems with different security levels

• Improper authentication and authorization mechanisms can expose internal data

• Changes to internal APIs may not undergo the same level of security testing as public-facing ones

• Easier to maintain compliance as they remain within the organization’s security perimeter

Integration with Legacy Systems

Risk Rating: 4 (Moderately High Risk)

AI systems that integrate with legacy healthcare IT infrastructure present elevated risk:

• The integration of multiple systems introduces new cybersecurity vulnerabilities, as demonstrated by incidents like the 2017 NotPetya malware attack that disrupted healthcare facilities through a third-party accounting software

• Legacy systems often lack modern security features and may have unpatched vulnerabilities

• Compatibility issues can create unexpected behaviors and security gaps

• Limited documentation of older systems increases risk during integration

• Many legacy systems weren’t designed with AI integration in mind

Integration of RPA with Generative AI

Risk Rating: 4 (Moderately High Risk)

The integration of Robotic Process Automation (RPA) with generative AI presents significant risks in healthcare environments:

• Combined technologies create powerful automation capabilities that can operate with minimal human oversight, potentially executing processes at scale before errors are detected

• RPA’s ability to interact with multiple systems using generative AI’s decision-making creates complex risk vectors that are difficult to monitor and audit

• Generative AI may produce plausible but incorrect outputs that RPA could automatically implement across systems without proper validation checks

• Potential for cascading failures when RPA acts on flawed generative AI outputs, affecting multiple downstream systems and processes

• The speed and scale of combined RPA-generative AI solutions can amplify small errors into significant operational or compliance issues

• Increased regulatory scrutiny as these combined technologies touch more sensitive data and critical processes

• Challenges in maintaining clear audit trails when generative AI’s reasoning isn’t fully transparent yet drives RPA execution

AI System Autonomy Attributes

Level of System Autonomy

Risk Rating: 1-5 (Based on degree of autonomy)

The risk increases with greater autonomy:

·        Level 5 (Highest Risk): Fully autonomous systems making decisions without human oversight

·        Level 4: Systems with minimal human oversight (only for exceptional cases)

·        Level 3: Semi-autonomous systems with routine human verification

·        Level 2: Advisory systems where AI recommends but humans make all decisions

·        Level 1 (Lowest Risk): Augmentation tools that only provide information to support human decisions

Agent Self Directed Agent Orchestration

Risk Rating: 4 (Moderatel High Risk)

AI systems that incorporate human oversight present moderate risk:

• Human oversight helps ensure AI decisions adhere to ethical standards and societal norms, reducing risk of bias and unintended consequences

• Human intervention provides important context, considers patient history, and interprets subtle nuances that may challenge machines

• Human-in-the-loop systems recognize that AI amplifies and augments rather than replaces human intelligence, focusing on improving efficiency and effectiveness of human interaction

• Still contains inherent AI risks but mitigated by human oversight

• Proper implementation requires clear accountability frameworks

Direct Patient-Facing AI Systems

Risk Rating: 5 (High Risk)

AI systems that interact directly with patients without healthcare provider mediation present high risk:

• Direct patient-facing systems may contain algorithmic bias from incomplete or unrepresentative training datasets, potentially exacerbating existing healthcare disparities

• Patient misinterpretation of AI outputs could lead to harmful actions

• Immediate clinical consequences if system provides incorrect information

• Higher regulatory scrutiny for patient-facing systems

• Requires additional usability and accessibility considerations

Provider-Only AI Systems

Risk Rating: 3 (Moderate Risk)

AI systems used only by healthcare providers present moderate risk:

• Professional interpretation helps filter and contextualize AI outputs

• Provider-only systems typically require human review before implementation of recommendations

• Clinical expertise provides an additional layer of validation

• Clinicians often experience mistrust of AI clinical decision support systems due to concerns about undisclosed risks and reliability

• Training requirements and adoption challenges can introduce implementation risks

AI System Interaction Attributes

Integration with Other AI Agents

Risk Rating: 4 (Moderately High Risk)

Healthcare AI systems that interact with other AI agents present elevated risk:

• When AI agents interact with multiple systems and external services, questions arise about data sharing practices and privacy protections

• As AI agents become more autonomous and integrated into clinical workflows, ensuring privacy, security and accessibility becomes paramount

• Complex interactions between multiple AI systems can be difficult to monitor and debug

• There is currently no standard way to flag AI-generated internet traffic as distinct from humans, making tracking and auditing difficult

• Accountability becomes diffused across multiple systems

Processing Sensitive Medical Imagery

Risk Rating: 4 (Moderately High Risk)

AI systems that process medical images present substantial risk:

• Medical imagery can potentially be re-identified when combined with other data sources [Nih]{.underline}

• Even when meta-data of images is removed, techniques may exist to extract identifying information from the images themselves [Nih]{.underline}

• High clinical impact if image analysis is incorrect

• Images often contain embedded metadata with patient identifiers

• Large file sizes create additional security challenges during transmission and storage

Access to External Data Sources

Risk Rating: 4 (Moderately High Risk)

AI systems with access to external data sources present significant risk:

• When AI systems can access external data, it raises concerns about data provenance, privacy, and potential re-identification of patients [Biomedcentral]{.underline}

• External data may not meet the same compliance standards as internal data

• The relationship between individuals and data collectors no longer follows traditional single-line connections, expanding into associations with multiple entities [Nih]{.underline}

• Integration with external sources creates additional attack vectors

• Difficult to validate quality and accuracy of external data

Real-time Decision Making

Risk Rating: 5 (High Risk)

AI systems making real-time clinical decisions present the highest level of risk:

• Immediate patient impact without opportunity for thorough review

• The speed at which AI systems operate may necessitate removing human oversight for time-critical decisions, which creates additional ethical and safety concerns [Cigionline]{.underline}

• System failures or incorrect decisions have direct clinical consequences

• Studies have shown that even experienced clinicians can struggle to consistently distinguish between accurate and inaccurate AI predictions and can be misled by inaccurate ones [Nsw]{.underline}

• Higher regulatory scrutiny for systems making real-time clinical decisions

Implementation and Governance Attributes

Level of Regulatory Compliance

Risk Rating: 5 (High Risk)

The level of regulatory compliance presents critical risk in healthcare AI:

• Healthcare AI must navigate complex regulations including HIPAA, FDA requirements, and FTC guidance, while regulations struggle to keep pace with AI development [Healthtech Security]{.underline}

• Various laws and guidance frameworks have been developed, including the White House’s Blueprint for an AI Bill of Rights, Biden’s executive order on AI, the Federation of State Medical Boards’ AI governance best practices, and WHO guidelines [Healthtech Analytics]{.underline}

• Different countries have varying data protection laws and ethical standards, creating a patchwork of protections [Nih]{.underline}

• Failure to comply can result in severe penalties, loss of licensure, and reputational damage

• Regulatory frameworks continue to evolve, requiring constant vigilance

Transparency and Explainability

Risk Rating: 4 (Moderately High Risk)

The degree of AI system transparency and explainability presents significant risk:

• Many AI models are ‘black boxes’ where only inputs and outputs are known, making them appear less trustworthy and difficult to explain how they develop their final results

• Explainability is essential for ensuring clinician trust and facilitating regulatory approval

• Lack of transparency creates risk for medicolegal and malpractice consequences

• Unexplainable systems make it difficult to identify and address biases

• Patients and providers have ethical right to understand how decisions are made

Organizational AI Governance Structure

Risk Rating: 4 (Moderately High Risk)

The presence and robustness of AI governance within healthcare organizations presents significant risk:

• Establishing effective AI governance requires executive leadership engagement, understanding of goals, and integration into existing clinical technology environments

• Only 16% of health systems currently have a systemwide governance policy specifically intended to address AI usage and data access

• An AI governance committee should include diverse members like healthcare providers, AI experts, and ethicists to consider different perspectives in decision-making

• Weak governance can lead to inconsistent implementation and monitoring

• Governance failures can result in higher regulatory exposure

Development and Testing Processes

Risk Rating: 4 (Moderately High Risk)

The rigor of development and testing processes for healthcare AI presents significant risk:

• Most AI models cannot easily generalize to other hospitals, countries or ethnicities, and most are not clinically validated on external or alternative data sets

• Insufficient independent validation is common, with many AI tools lacking peer-reviewed studies confirming clinical effectiveness

• Healthcare AI requires continuous monitoring after deployment to identify adverse events using effective post-market surveillance

• Inadequate testing can lead to patient harm and liability

• Testing methodologies for AI are still evolving, especially for generative AI