Building an AI Clinical Documentation System: Ambient Scribe Architecture for Founders

The pitch for an AI clinical documentation product is easy to make and hard to build. A clinician talks to a patient, the software listens, and a finished, coded, EHR-ready note appears. The demo lands in thirty seconds. The system behind it is a pipeline with at least six stages, each of which can fail in a way that produces either a wrong note or a HIPAA incident.

This post walks through the architecture we use for ambient scribes and clinical documentation systems, the decisions that matter most, and where founders and CTOs tend to underestimate the work. It is a companion to our overview of what we build for AI clinical documentation. This guide goes a layer deeper on the engineering.

The pipeline, stage by stage

An ambient scribe is not one model call. It is a sequence: audio input → speech-to-text (ASR) → LLM summarization → structured output → clinician review → EHR write-back. Treating it as a monolith is the first mistake, because each stage has a different failure mode, a different latency budget, and a different compliance boundary.

Audio capture and speech-to-text

The recording captures a real clinical encounter: two or more speakers, medical terminology, interruptions, and background noise. Off-the-shelf consumer transcription struggles with drug names, dosages, and specialty vocabulary, which are the words that matter most. You have two broad options, and compliance drives the choice as much as accuracy.

The first is a HIPAA-eligible commercial ASR service under a Business Associate Agreement, for example AWS Transcribe Medical or Deepgram where a BAA is available. The second is a self-hosted model such as Whisper, deployed inside your controlled environment, so the audio never leaves infrastructure you govern. The trade-off is real: managed services ship faster and score better out of the box; self-hosting gives you tighter data control and lower marginal cost at scale, at the price of running the inference yourself. Neither is universally correct; pick the one that fits your data posture and budget.

LLM summarization into structured notes

Once you have a transcript, the summarization step turns free-form conversation into a structured clinical note, most commonly the SOAP format (Subjective, Objective, Assessment, Plan). Founders assume a single prompt will do; it tends to fall short.

Specialties structure notes differently. A primary-care visit, a cardiology follow-up, and a dermatology exam do not share the same note skeleton. The durable design is a templating layer: the LLM fills a specialty-specific structure rather than freely composing prose. New specialties then become new templates, not new models or full rebuilds. This keeps the system extensible and, just as important, keeps output predictable enough to review quickly.

Coding assistance: assistive, never automatic

Many documentation products add ICD-10 diagnosis coding and CPT procedure coding to produce billing-ready output. This is valuable and risky in equal measure. Coding drives reimbursement and touches fraud-and-abuse exposure, so the design principle is non-negotiable: coding is assistive. The system proposes codes; a clinician reviews and finalizes before anything is submitted; and every step lands in an audit trail. Automatic submission of AI-suggested codes is a liability.

The clinician review gate

The most important stage is the one demos skip. A finished note is a draft until a clinician reviews and signs it. Your architecture should make the review step fast and unavoidable: draft → clinician edit → finalize → post. The reviewing clinician is legally and clinically accountable for the note, so the UI has to surface uncertainty rather than hide it, and the audit trail has to record who changed what and when.

EHR write-back

The last mile is posting the finalized note back into the chart. In practice this means integrating over FHIR R4 or HL7 v2.x with the major EHRs: Epic, Cerner / Oracle Health, Athenahealth, eClinicalWorks. This is where timelines slip, because each EHR integration has its own onboarding, sandbox access, and quirks. If an EHR does not expose a clean write path, the honest answer to a founder is that a workaround exists and costs time.

Where PHI lives, and where it must not

Protected health information flows through every stage of this pipeline: the raw audio is PHI, the transcript is PHI, the note is PHI. The controlling rule: PHI only touches services that are HIPAA-eligible and covered by a Business Associate Agreement. For LLM processing that means routing through providers like Azure OpenAI or AWS Bedrock under a BAA, or self-hosting where no BAA option exists. That boundary belongs in code and infrastructure, not in a configuration checkbox someone can flip off.

Alongside that, the standard HIPAA technical safeguards apply throughout: encryption at rest and in transit, role-based access controls, immutable audit logging, and keeping PHI out of application logs and URLs. We go deeper on how we engineer these in our write-up on HIPAA-compliant software development, including why "HIPAA-certified" is a phrase to treat with suspicion: HIPAA is a regulation, not a certification.

The regulatory questions you cannot answer with engineering alone

Two questions come up on every clinical documentation build, and both need advisors, not just architects.

The first is FDA Software as a Medical Device (SaMD) classification. Ambient scribes are generally advisory and often fall outside Class II SaMD, but classification depends on intended use, and that determination belongs to your FDA regulatory consultant. We architect so that a SaMD path is possible if you need it; we do not make the legal call.

The second is state medical-board rules on AI-assisted documentation, which vary by state. We build in clinician-review gates and disclaimer language so the boundary between software and clinical judgment stays explicit; the applicable board requirements stay with your regulatory advisor.

Related building blocks

Clinical documentation rarely ships alone. If your product also answers clinical questions from guidelines or literature, the retrieval side of the house is its own discipline; see healthcare RAG and clinical knowledge retrieval for how grounding and citations are engineered so the assistant does not invent clinical claims.

How to scope the build

If you are starting fresh, a focused Project Build (capture, transcription, SOAP generation, coding, and one EHR write-back) is a defensible first scope; additional specialties and integrations are then additive rather than a rebuild. If you already have something running and want an honest read on its ASR/LLM pipeline and its HIPAA and BAA posture, a Tech Audit is the faster path. To sanity-check budget before committing, our cost estimator gives a rough shape.

This is the broader discipline of AI engineering for healthtech: the demo is the easy part, and the difference between a prototype and a product is the review gate, the compliance boundary, and the EHR last mile. If you are building in this space, talk to a founder: no account managers, a direct conversation about your pipeline and your PHI flows.