HIPAA-Compliant LLM Access Is Not That Hard

HIPAA-compliant LLM access really is not that hard.

I came to this conclusion after going fairly far down the wrong path.

It started with Kimi K3 and GLM-5.2. These open-weight models are getting close enough to proprietary frontier models that they need to be taken seriously.

My first thought was that healthcare companies could run models like these themselves and keep patient data inside their own environment.

That is an appealing idea. No protected health information sent to a model API. No dependence on a third-party inference service. A company controls the hardware, the model weights, the network, and the logs.

Then I looked at the hardware.

These are not small models. Kimi K3 has 2.8 trillion parameters, and the vLLM project describes an eight-NVIDIA-B300 deployment as its easiest way to run the model. GLM-5.2 has 753 billion parameters; NVIDIA’s current quantized release targets Blackwell hardware and supports runtimes such as vLLM and SGLang.

The exact footprint depends on the precision, context length, concurrency, and serving stack. That is the point. An organization considering this path needs to budget for more than a GPU server: high-speed GPU interconnects, storage for weights and logs, redundant infrastructure, monitoring, patching, model serving, and someone who can operate all of it. For serious production traffic, that can become a large infrastructure project quickly.

That does not make local inference impossible. For the right workload, volume, and organization, it may be a reasonable choice.

But it makes “we should run our own frontier model because HIPAA” a much more expensive sentence.

The local-model path has more than one vendor

The next option is a GPU cloud. Some providers will sign a business associate agreement (BAA), which is a necessary part of handling protected health information with a service provider.

But the details matter.

A BAA may cover specific compute, storage, and logging services rather than everything the vendor offers. The organization still needs to understand where prompts, responses, embeddings, backups, traces, and support data go. “The GPU is covered” is not the same as “the whole system is covered.”

That is real diligence, but it is also not unique to open models.

Then there is the more obvious question: what about the clouds most healthcare companies already use?

AWS lists Amazon Bedrock as HIPAA eligible. Microsoft offers a BAA for in-scope Azure services, and its Azure AI Foundry documentation describes its HIPAA compliance offering. Google supports HIPAA workloads through Vertex AI.

So a healthcare company can already get managed access to capable models from AWS, Microsoft, or Google without buying a GPU rack or operating an open-weight model.

The model is available. The work is deciding what to do with it.

HIPAA is a system property

A BAA does not make an application safe by itself. Neither does a local GPU server.

The company still needs appropriate access controls, encryption, audit logging, retention rules, and an architecture that keeps PHI inside covered services. It needs to know which users can ask which questions, which records the system can retrieve for them, and whether sensitive data is leaking into an observability tool or a debugging log.

Those are not optional details around the LLM. They are the system.

A useful healthcare assistant also needs to be more than a chat box pointed at a model. An end user should see an authenticated application that retrieves only the records they are allowed to access, shows where an answer came from, and makes it easy to correct or escalate an uncertain result.

Take prior authorization. An LLM could help assemble information from a patient record into a draft packet. That may save a person time. But somebody still needs to decide what information is relevant, validate that the draft is correct, submit it through the appropriate channel, and handle the exception when the case does not fit the usual pattern.

The same is true for chart summarization, intake, coding support, patient-message triage, and internal policy search. The question is not just whether a model can produce a plausible answer. It is whether the workflow makes a person faster without making a mistake harder to catch.

Start with the workflow

I suspect plenty of healthcare companies are treating HIPAA as the blocker when the larger issue is that they have not identified a narrow enough problem worth solving.

“Give our staff an LLM” is not a workflow.

“Help the prior-authorization team find the relevant clinical history, draft a packet, and flag missing information for review” is much closer. It gives the team something to evaluate: time saved, completeness, error rate, review burden, and the cases where the system should stop and ask for help.

Once that workflow is clear, the infrastructure decision becomes more practical.

A managed model service may be the right answer if it fits the company’s covered environment and the team wants to focus on the application. A locally hosted open model may be the right answer when the organization has unusually strict control requirements, enough sustained volume to justify the infrastructure, or a reason to operate the model as a core capability.

Neither option removes the need for careful design. Both can be part of a HIPAA-compliant system. Both can also be used carelessly.

The LLM is not really the hard part.

The hard part is picking a useful workflow, connecting the right data, evaluating the output, and deciding where a human needs to remain involved.

What healthcare workflow would you be comfortable giving an LLM access to today?