← Back to blog

Compliance

How to Run a HIPAA-Compliant LLM: Four Deployment Paths

There are four ways to put an LLM in front of patient data, and they differ in what you take on. A comparison of cost, model access, and compliance burden

Chris Williams, MD

TL;DR

There are four compliant ways to run an LLM on patient data: a hyperscaler BAA, a direct vendor BAA, a BAA-backed open-weight API, or self-hosting. They differ mainly in how much infrastructure and compliance work you take on, and which models you can reach.

There is no such thing as a HIPAA-compliant model. There are compliant deployments, and in practice they come in four shapes.

The choice between them is less about technology than about how much of the compliance burden you want to own, and how much of the model catalog you are willing to give up.

Path 1: A hyperscaler BAA

You use Azure OpenAI, AWS Bedrock, or Google Cloud's Vertex AI, and you rely on the cloud provider's own business associate agreement.

  • What you get. An agreement that is already in place for covered entities, managed infrastructure, and a well-trodden security review path.
  • What you take on. Configuration, third-party compliance verification, annual pen-testing. Microsoft notes that its BAA is available through the Product Terms and that "there is no separate contract to sign" - but also that using Azure "doesn't automatically impart compliance onto your cloud solutions." AWS makes the same point: "There is no HIPAA certification for a cloud service provider."
  • When it fits. Your stack is already on that cloud, and you need frontier closed models.

Path 2: A direct vendor BAA

You contract directly with a model vendor - OpenAI's API under Modified Retention, or Anthropic's first-party API under HIPAA readiness.

  • What you get. Direct access to frontier closed models with an explicit agreement, and in OpenAI's case a self-serve flow.
  • What you take on. Scope management. OpenAI's API eligibility is "contingent on Customer's account being provisioned with Modified Retention," and Anthropic's BAA explicitly excludes Free, Pro, Max, and Team plans along with the Console and beta features.
  • When it fits. You need a specific closed model and can live inside the covered surface.

Path 3: A BAA-backed open-weight inference API

You use a provider whose product is the compliance posture - a unified API over open weights, with the agreement attached to the service.

  • What you get. A single agreement covering the inference path, access to current open-weight checkpoints, and no cloud infrastructure to stand up. All subject to a minimum annual spend in return for signing a BAA.
  • What you take on. Vendor diligence. You are relying on a newer provider, so the agreement's scope, the subprocessor list, and the retention posture deserve real reading rather than a glance.
  • When it fits. You want the unified-API pattern, you prefer open weights, you do not want to run GPUs, and you have a large enough budget to cover the minimum spend of these vendors.

Path 4: Self-host open weights

You run the model yourself, inside your own environment.

  • What you get. Maximum control. PHI never leaves your boundary, and the vendor relationship disappears entirely for inference.
  • What you take on. Everything else. GPU capacity, inference optimization, availability engineering, patching, and the full technical safeguard set on infrastructure you operate. You also own the model quality problem: serving a frontier open weight well is a specialized job.
  • When it fits. You have an existing GPU footprint, an SRE team, and a requirement that data never leaves your network.

Side by side

Hyperscaler BAADirect vendor BAABAA-backed APISelf-host
AgreementBundled in cloud termsVendor-specificVendor-specificNone needed for inference
Closed model accessSomeYesNone todayN/A
Newest open weightsLagsN/AFirstYou choose
Infrastructure workSignificantMinimalMinimalSubstantial
Marginal cost profileHighestHighLowFixed plus ops
Compliance surface you ownMediumLowLowHigh

How to choose

Work down this list and stop at the first honest yes.

  1. Must the data never leave your network? Then Path 4, and budget for the team it requires.
  2. Do you need a closed frontier model specifically? Then Path 1 or 2, and accept the per-token cost.
  3. Do you need the newest open weights? Then Path 3, because hyperscaler catalogs lag new checkpoints.
  4. Otherwise? Path 3 for cost and simplicity, or Path 4 if you already have the infrastructure.

The mistake I see most often is choosing Path 4 for control and then discovering that the operational load is a full-time role nobody was hired for. The second most common is choosing Path 1 for convenience and then finding the latest model you actually want will not be in the catalog for another 6 months.

What every path needs regardless

The agreement is the entry ticket, not the program. Whichever path you pick, you still need:

  • The technical safeguards - access control, audit controls, integrity, authentication, transmission security.
  • A defined retention posture, and a documented default.
  • An internal owner for all of the above, because configuration drifts.

That list is identical across all four paths. Only the division of labour changes.

Common failure modes by path

Each path has a characteristic way of going wrong.

  • Hyperscaler BAA: configuration drift. The agreement exists, you're happy with the model catalog. Then the latest hot model comes out that won't be available on your hyperscaler for 6+ months. Your AI team gets more and more frustrated that they're not working with frontier models. And you have to go through the whole vendor procurement process all over again.
  • Direct vendor BAA: surface confusion. A team enables HIPAA support on the API, then PHI routes through a consumer plan or the console, which the agreement excludes. The covered surface is narrower than the product's surface area.
  • BAA-backed API: thin diligence. The agreement is signed but you need 100% up-time with sub-300 ms latency. And the provider you chose doesn't meet those needs.
  • Self-hosting: unowned operations. The cluster works in the pilot. Eighteen months later the person who built it has left, the model is two generations behind, and nobody owns the patch cadence.

None of these is exotic, and all four are cheaper to prevent than to remediate. The common thread is treating compliance as a procurement event rather than an operating practice.

Migrating between paths

Teams rarely pick once. The common sequence is self-hosting or a hyperscaler first, then a managed API once the operational cost becomes clear - or the reverse, when a contract requires data to stay inside a network.

Two things make migration survivable. Keep the model call behind your own abstraction rather than scattered through the codebase, so switching providers is a configuration change. And keep the compliance artifacts - agreement, subprocessor list, retention configuration - in one place, because a security review will ask for all of them at once regardless of which provider is current.

OpenMed Router is the unifying path: a signed BAA covering the open-weight model catalog, zero-retention isolated inference, and AES-256 encryption in transit and at rest.

Still deciding which path fits? Join the waitlist.

Chris Williams, MD

Chris Williams, MD is a physician, clinical AI researcher and the co-founder of OpenMed Router, working to make open source AI models safely accessible to healthcare organizations under HIPAA. He writes about clinical AI, model selection, compliance, and the practical adoption of open source inference in clinical and operational workflows.

Join the waitlist

Be first in line for HIPAA-compliant open source inference