Overview
Responsible enterprise AI for operations and knowledge work
A multi-site health system deployed grounded assistive AI over approved content, with role-based access, human escalation, and continuous evaluation—reducing handle time and search friction while keeping clinical and compliance boundaries explicit.
Contents
At-a-glance
Client (illustrative): Unity Health Network—acute and ambulatory care across several regions. Engagement model: discovery and pilot (internal operations, access center, and parts of health information management), then phased expansion with information governance and privacy sign-off for each content domain.
Executive summary
Leaders needed to reduce administrative burden and time spent searching policies, payor rules, and internal playbooks, without introducing uncontrolled generative use of EHR or billing data. Public large language models were ruled out for regulated flows; the organization required retrieval-grounded answers, citation to source documents, identity-aware entitlements, and a clear “AI suggests; human decides” line for any action that could affect care or money.
Business case and operating context
Stakeholders and success criteria
Sponsors included Chief Information Officer (platform owner), Chief Medical Information / Quality (clinical appropriateness), and Privacy & Compliance (HIPAA-style obligations and policy alignment). Success metrics combined operational KPIs (e.g. average handle time, first-contact resolution where applicable) with trust measures: staff-reported ease of finding policy, and percentage of model answers with verifiable inline citations to vetted sources.
Drivers
- Fragmented search across PDFs, intranet, and shared drives
- High training cost for seasonal policy and payor changes
- Risk of inconsistent answers in access center and back-office
- Pressure to explore AI without bypassing record or consent rules
Intended value
- Faster, cited answers for internal policy and operational questions
- Structured handoffs to humans for edge cases and exceptions
- Audit trail: who asked, what was retrieved, what was shown
Technical design and guardrails
Retrieval and generation
NeuralFaaruuq implemented a RAG pattern: document ingestion from approved corpora (policies, benefits summaries where allowed, standard operating procedures), chunking with overlap, embedding index, and re-ranking for top-k passages. The model receives only retrieved excerpts plus a constrained system prompt describing role, disallowed uses, and citation requirements. Where EHR text is out of scope, metadata-only or redacted extracts are used; free-form clinical summarization from production records was not enabled in the pilot.
Identity, entitlements, and integration
Access to corpora and features is mapped to the existing IdP and groups (e.g. access center vs. HIM). The assistant is surfaced in a web experience and, where approved, in the ticketing / CRM sidebar with the same entitlements. API keys are scoped per environment; no client data is used to train foundation models in this program.
Evaluation and release discipline
Before each wave, we ran offline eval sets: gold Q&A from subject-matter experts, adversarial “trick” questions, and regression tests when documents change. Online metrics include citation rate, user thumbs feedback, and escalation rate to human support. Staged rollout by department limited blast radius while playbooks matured.
Capability map (simplified)
| Layer | Function | Notes |
|---|---|---|
| Content pipeline | Ingest, version, retire | Source-of-truth tagging; legal/privacy sign-off per corpus |
| Vector store | Semantic search | Region pinned; encryption at rest; access logs |
| Orchestration | Prompt + tool routing | Refusal templates; max tokens; PII filters |
| Model API | Completion | Org keys; no training on UHN prompts (contractual) |
| Observability | Audit, dashboards | Request ID, user, corpus, retrieved chunk IDs |
Governance, safety, and assurance
Change control: new documents or major revisions trigger re-indexing and a short re-validation window. Incident process: if a harmful or noncompliant output is reported, the pipeline can disable a feature flag for a cohort while retaining logs for review. Training: staff were trained on appropriate use, escalation, and the difference between AI-generated suggestions and system-of-record actions.
Outcomes and next phase
Business and experience
- Meaningful reduction in average handle time in pilot access-center cohorts (exact % under client NDA)
- Improved staff-reported ease of finding policy vs. legacy search
- Clearer, faster onboarding to seasonal policy changes with cited excerpts
Risk and technology
- High citation coverage in pilot; continuous monitoring for drift when sources update
- Escalation paths preserved; no fully automated clinical or financial decisions
- Next phase: expand corpora, optional language locales, and deeper ITSM integration
Conclusion
The most durable gains came from pairing a disciplined content and retrieval foundation with executive clarity on what AI is allowed to do. Incremental “chat everywhere” without corpus ownership and evaluation would not have met safety or audit expectations. NeuralFaaruuq remains engaged for roadmap delivery and evaluation harnesses as Unity Health Network scales the program.
