Shadow AI in healthcare is no longer a fringe behavior. It’s becoming prevalent.
Studies indicate that healthcare professionals are increasingly using generative AI tools such as ChatGPT, Google Gemini, Microsoft Copilot, and other healthcare-focused tools, particularly for medical research summarization and clinical documentation. A recent survey found that healthcare professionals increasingly use unauthorized AI tools at work, including for direct patient care. This highlights the need for clear policies, enterprise-wide guardrails, and rigorous monitoring.
According to IBM’s 2026 Cost of a Data Breach Report, security incidents involving shadow AI have more than doubled. Among organizations that experienced AI-related breaches, 92% lacked proper AI access controls, and 68% had no AI governance policy in place.
For healthcare organizations, the stakes are even higher:
- Healthcare remains the costliest sector for data breaches, with an average breach cost of $6.64M in 2026, despite a 10.5% decline from $7.42M in 2025. The sector has remained the costliest for 13 consecutive years, underscoring the financial impact of protecting sensitive patient data across EHRs, billing systems, and other connected platforms. (Source: IBM Cost of a Data Breach Report 2026)
- American Medical Association’s 2026 Physician Survey on Augmented Intelligence, published March 12, 2026. The survey of 1,692 physicians found that AI is increasingly being used for tasks such as summarizing medical research, creating clinical documentation, and generating care plans. Without clear guardrails, these AI-enabled workflows can introduce new risks to the healthcare attack surface.
- The global meantime to identify and contain a breach rose to 247 days in 2026, up from 241 days the previous year. For healthcare organizations, where sensitive patient data is spread across EHRs, billing systems, and other connected platforms, longer breach lifecycles can increase the opportunity for exposure and amplify the cost of response. (Source: IBM Cost of a Data Breach Report 2026)
The Regulatory Risks Associated with Shadow AI in Healthcare
In a heavily regulated industry like healthcare, shadow AI creates compliance risks under HIPAA, Section 1557 of the Affordable Care Act, and emerging state-level AI laws.
- HIPAA rules require vendors handling PHI on behalf of a covered entity to sign a Business Associate Agreement (BAA) to receive, store, or process AI. Using AI tools to process PHI without appropriate contractual safeguards can attract monetary penalties and even criminal prosecution in some cases.
- Section 1557 of the Affordable Care Act prohibits bias and discrimination in healthcare programs and activities. Organizations using AI to automate decision-making must assess these systems for potential discriminatory outcomes.
- State-level AI regulations include specific AI requirements for healthcare. In Texas, for example, the Texas Responsible Artificial Intelligence Governance Act (HB 149), effective January 1, 2026, requires providers to disclose their use of AI in patient care clearly and in plain language. Separately, SB 1188, effective September 1, 2025, requires practitioners to review AI-generated records against Texas Medical Board standards and to disclose their AI use to patients.
Top AI-related Incidents that Can Trigger Compliance Violations
- Personal Account Use: Research shows up to 71% of healthcare workers have been found to use personal AI accounts for work. Employees sharing PHI into personal, unapproved AI accounts can create HIPAA Privacy and Security Rule risks if the organization cannot control access, use, disclosure, or security of that PHI.
- Data Retention by Unsanctioned Tools: If an AI provider retains or uses PHI beyond the authorized purpose, this can create HIPAA concerns around permitted uses/disclosures and, where applicable, Business Associate Agreement (BAA) requirements.
- Silent Third-party Sharing: Chatbots and appointment scheduling assistants that share PHI, such as appointment details and symptoms, with analytics or other third-parties without appropriate authorization/safeguards can again create HIPAA Privacy Rule violations and potential BAA issues.
- Blind Spots in Automated Workflows: When organizations fail to assess AI-enabled workflows, they may not know where PHI is processed, stored, or shared. This can create HIPAA Security Rule risks when AI-related vulnerabilities are not identified and appropriate safeguards are not implemented.
These incidents don’t just create AI governance risks; they can directly expose healthcare organizations to regulatory violations when AI systems process protected health information without appropriate authorization, safeguards, or oversight.
Checklist for Leaders to Tackle Shadow AI in Healthcare
Closing the shadow AI gap requires a structured approach. Leaders must:
- Build clear AI usage policies: Establish clear rules around AI use in healthcare, defining what is permitted, restricted, and prohibited in simple, accessible language.
- Identify purpose-built AI tools aligned with organizational goals: Providing approved, fit-for-purpose AI tools can help reduce reliance on unsanctioned alternatives while giving clinicians efficient tools to support their work.
- Deploy technical guardrails to monitor AI use: Network-level visibility into AI tools, controls that flag PHI leaving the environment, and dedicated AI monitoring solutions can help organizations detect and enforce policy violations.
- Clearly communicate policies and provide practical training: Make expectations clear and reinforce them with real-world scenarios, such as drafting a discharge summary or transcribing a patient visit.
- Emphasize AI literacy: Help staff understand why certain actions are prohibited, the risks they create, and how sensitive data can be exposed when entered into public or unapproved AI tools.
- Continuously govern and gather feedback: Shadow AI is an evolving challenge. Healthcare organizations must continuously monitor AI use, incorporate feedback, and update policies and controls as new tools and use cases emerge.
When these steps are missing, the consequences show up fast in healthcare settings:
- Hallucinated outputs presented with a confident tone and without clear indications of uncertainty
- Complex audit trails, making it difficult to trace AI-related incidents and identify root causes
- Loss of patient trust, along with potential privacy, financial, and identity-related risks
- Misdiagnoses and inaccurate recommendations resulting from unreliable or inappropriate AI outputs
- Eroded accountability and legal or regulatory repercussions
KANINI’s Shadow AI Risk Prevention Framework
Explore Propel SDLC – KANINI’s AI Orchestration Framework for Enterprise SDLC.
Ready to Build a Resilient, AI-First Healthcare System?
The risks of ungoverned AI are real. But with the right guardrails, so is the opportunity.
“‘The healthcare organizations we work with aren’t trying to ban AI — they’re trying to catch up to how their clinicians are already using it,’ says [Name, Title, Kanini].
Having built data infrastructure across EHRs, claims, and clinical systems for healthcare clients, KANINI sees where shadow AI typically enters, and where governance needs to start. With the right planning and a trusted AI partner, healthcare organizations can move from ungoverned risk to sanctioned, compliant AI that improves productivity, automates workflows, and strengthens patient care.
KANINI offers:
- Structured audit of PHI exposure
- Gen AI-powered Clinical Decision Support Systems
- HL7/FHIR-compliant foundation for data
- Common Data Framework for Healthcare, and more.
Author

Jatinder Bedi
Jatinder Bedi is a Data and Artificial Intelligence leader at KANINI, with expertise in Data Science and AI. A Data Science Architect and AI scholar, he is passionate about advancing AI and has championed AI practices across industries and start-ups throughout his 17+ years of experience.


