Vehicle insurance claims start with a claimant photographing their own damage and an adjuster interpreting whatever arrives. We built both halves of that handoff for an insurance AI startup: an on-device app that tells claimants when a photo is usable before they submit it, and a detection pipeline that reads the resulting images into structured part-and-damage findings for the adjuster.
Claims submitted through the guided capture app needed follow-up photos 20.3% of the time, against 29.7% for freely taken photos. Carriers running the detection pipeline as a pre-processing step saw claim handling time drop by roughly 15%.
The problem
Adjusters review claim photos by hand to establish what is damaged, how badly, and what it costs to put right. That review is expensive and slow, and the claimant waits through all of it.
Automating it runs into a problem upstream of any model: the input is a stressed person in a parking lot with a phone. Glare, motion blur, poor exposure, and photos that simply don’t contain the damage are all normal. Worse, a low-quality image doesn’t just fail — it invites a model to read an artifact as a dent.
So the work split in two. Fix the capture, then read the capture.
Part one — getting a usable photo at the point of capture
We built a mobile app that assesses each frame in the viewfinder against three conditions: is a vehicle present, is damage visible, and is the image itself good enough to use.
Vehicle presence and damage presence run as two independent CNNs. Image quality is assessed separately, combining pixel-level and filter-based measures with models trained to spot obstruction, glare, blur, and exposure problems. Quality turned out to matter at least as much as content — a well-framed photo of the right damage is still useless if nobody can tell what they are looking at.
When all three conditions are met the shutter is enabled; when they aren’t, the app says which one failed and what to change.
The constraint that shaped everything. This had to run in real time on whatever phone a claimant happened to own. Phone GPUs exist but vary enormously, so we could not depend on one being usable. That ruled out anything heavy: models were kept small enough to run on CPU, converted to TensorFlow Lite, and the surrounding logic rewritten in Java and Swift so the whole assessment loop stayed on-device.
Part two — reading the claim
Submitted images run through detection models built on RetinaNet, Mask R-CNN, and YOLO architectures, trained on data labelled specifically for vehicle parts (front bumper, headlights, door panels, rear fender, and so on) and damage types (dents, scratches, cracks, missing pieces). Outputs are thresholded and filtered, then damages are associated to parts: dent on the front bumper, headlight partially missing.
The hard part is not detection, it’s reconciliation. Claimants submit one to five photos per damage. Treat each image independently and a single dented bumper becomes five dents, and the estimate is nonsense. Damages are matched across images so each real-world damage is counted once.
That aggregate is then reconciled against the claim’s text: does the described incident match what the photos show, is something documented but not photographed, is there damage in the photos nobody mentioned. The system also infers what it cannot see — likely internal damage, or signs that pre-existing damage is being folded into a new claim.
Finally, per-part severity drives a repair-versus-replace call and a price, adjusted for vehicle make and model and for local part and labour costs, since that varies enough by market to flip the decision.
Where the human stays in the loop
Output pre-fills the carrier’s existing claim forms. The adjuster’s job becomes reviewing and correcting rather than starting from a blank form and a folder of photos.
The system is scoped to low and medium severity damage and knows it. Where damage is severe, or where there are strong indications of internal damage, the response flags the claim for escalation rather than producing an estimate — those claims turn on factors that aren’t in the photographs. Adjusters get their time back on exactly the cases where judgment is needed.
Corrections feed back as labels. Carriers supply annotations for claims where the models underperformed, which drive retraining.
Results
Guided capture, first carrier deployment (India, January 2018): claims with app- captured photos required additional photos 20.3% of the time, versus 29.7% for freely captured photos. Fewer follow-up requests means faster settlement, and higher baseline image quality meant more claims could be decided without an in-person inspection. Claimants reported finding the corrective guidance genuinely helpful during what is, by definition, a bad day.
Damage detection, first carrier deployment (US, February 2018): approximately 15% reduction in claim handling time versus an adjuster working from scratch. The same architecture and workflow were subsequently localized for carriers in other markets, adapting vehicle types and cost models.
Detection performance envelope. Correct part identification 86% of the time, correct damage-to-part association 79%. Accuracy is markedly better on large standard panels and degrades on small, detailed components. These figures describe a first-pass assistive system whose output an adjuster reviews — not an autonomous estimator.
A secondary benefit the carriers found for themselves: because every claim now carries structured tags for parts, damage types, and severity, they could filter and retrieve claims by property for analysis. Previously that took manual inspection and trial and error.
What it doesn’t do
- It doesn’t settle claims. Every output is reviewed; severe and internal-damage cases are routed away from the system entirely.
- It doesn’t assess high-severity damage. Out of scope by design.
- Small detailed parts remain weak. Reflected in the 86% figure and known to users.
- The capture app can’t fix a bad camera. It rejects unusable frames; it doesn’t enhance them.
Stack
Detection: RetinaNet, Mask R-CNN, YOLO. On-device: TensorFlow Lite, native Java and
Swift. TODO — training infrastructure, serving setup, how carrier localization was
managed.