A medical billing company’s audit team could only review a small fraction of the encounters they coded each day, and they picked that fraction at random. We built a set of models that score every encounter for the likelihood of a coding error, so the same team reviews the same number of encounters and finds considerably more of the ones that are wrong.
In production, randomly selected encounters carried an error rate of 4.21%. The encounters our system selected carried 6.05% — a 1.84 point lift, or 43% more errors surfaced for the same auditor hours. The system has run continuously since February 2019.
The problem
US medical facilities convert clinical notes into standardized CPT codes before submitting to insurers. Getting those codes right matters in both directions: under-coding costs the provider money it is owed, while over-coding or mis-coding exposes them to clawbacks and penalties if an external audit finds a pattern of misrepresentation. Billing operations therefore audit their own coders continuously, aiming to hold accuracy in the mid-to-high nineties.
Our client’s audit team pulled a random sample of each day’s encounters. That sample was statistically sound for measuring coding quality, but inefficient at improving it — most encounters reviewed were correct, so most of the review capacity produced no correction.
Two things made this harder than a standard classification problem.
Coding involves genuine judgment. Two experienced coders can read the same chart and reach different defensible codes. “Error” is not a clean label sitting in the data; it is an auditor’s determination. Any model here is predicting what an auditor would conclude, not recovering ground truth, and the system had to be designed and evaluated on that basis.
Almost nothing was labelled. Because only a sampled subset was ever audited, the overwhelming majority of encounters had no verdict attached — and the labelled subset was small enough that training directly on it would have overfit immediately.
What we built
We started by sitting with the audit team and decomposing what they actually do. Rather than one model predicting “is this wrong,” we mapped the distinct failure modes and the lookup each auditor performs to confirm them. Three types accounted for the bulk of real corrections:
- Errors in the CPT code itself
- Errors in the diagnosis or diagnosis codes
- Missing procedure codes or modifiers
Each got its own model, checking the coded output against what the underlying record supported.
Turning the unlabelled data into an asset. The large unaudited corpus was the obvious place to get signal, so we pretrained a base network on it with a self-supervised objective: given an encounter’s text, detect whether sentences had been manipulated. This is a simplified variant of the masked-language-modelling task used to pretrain large language models, chosen to fit the size of the corpus and the compute available. The result was a base model with a working representation of what a normal, internally consistent encounter looks like — learned entirely without audit labels.
We then transfer-learned that base separately for each error type, using the small labelled set for the supervised stage only. Each model emits a probability; the scores are combined into a single error score per encounter.
Routing, not deciding. Each morning the audit team submits the previous day’s coded encounters along with how many they have capacity to review. The system ranks by error score and returns the top slice. Auditors confirm or reject each flag and correct what needs correcting — the determination stays with them, and their corrections come back as labels for the next retrain.
Critically, the client kept a random sample running alongside the model-selected queue. That control group is why the numbers below exist at all: it measures the underlying error rate continuously, so the system’s lift can be verified in production rather than assumed from a validation set.
The system changes the thing it measures
Coding drift was expected and is routine. What was less obvious at design time is that deploying the system altered the behaviour it was trained on.
Once coders knew which patterns drew audits, the patterns shifted. Once auditors had a ranked queue, their review behaviour shifted too. The distribution the models were fitted to is partly a product of the models’ own presence, which means performance decays for reasons no amount of held-out validation would predict.
We handle this with quarterly retraining on accumulated auditor verdicts, so the models track the current state of coder and auditor behaviour rather than the 2018 version of it. In a system where a deployed model changes its own input distribution, retraining cadence is not maintenance overhead — it is the design.
Results
| Random selection | Model selection | |
|---|---|---|
| Error rate in reviewed encounters | 4.21% | 6.05% |
| Errors surfaced per 1,000 reviewed | 42 | 61 |
Same audit headcount, same number of encounters reviewed, 43% more errors found and corrected before submission. Corrections run in both directions: recovered under-billing the client was entitled to, and caught mis-billing before it reached an insurer and became an external audit finding.
The first version shipped in February 2019, two weeks ahead of schedule and within budget. It met the success criteria set for the pilot and converted into an ongoing engagement covering maintenance, retraining, and model updates. Since then the relationship has expanded into other parts of the client’s operation, including provider and customer information changes and insurance dispute handling.
What it doesn’t do
- It doesn’t determine whether a code is wrong. It ranks encounters by the likelihood an auditor would find something. Every verdict is human.
- It doesn’t reduce audit headcount. It reallocates existing capacity. The gain is in what that capacity finds, not in spending less on it.
- It doesn’t replace the random sample. The control group has to keep running, both to measure baseline error and to catch error types the models don’t score.
- It’s tuned to this client’s coders and specialties. The models are fitted to emergency and urgent care coding as this team practices it. Another organization would need its own labelled history.
Stack
TODO — architecture family for the base network, serving setup, batch scheduling,
monitoring.