Case study · complaint routing · public data
Same model. Half the mistakes.
We trained a small open AI model to sort real consumer complaints, then tested it on 1,190 it had never seen. Before training it filed 41.6% of them right. After, 72.0%. Same model, same instructions, same test.
362 more complaints landed in the right place, out of 1,190.
The case in five lines
Drafted turns specialized human expertise into training signal enterprise AI teams can ship.
- The challenge
- Off the shelf, a small open AI model sorted real consumer complaints into the right product and issue only 41.6% of the time. Too many mistakes to trust with a real queue.
- The Drafted solution
- We trained a small adapter for an open Qwen3-8B model on 5,514 real, labeled complaints, without changing the base model.
- What we delivered
- The full loop, end to end: data split, cloud training, and a held out test on 1,190 complaints the model had never seen, scored the same way before and after.
- The outcome
- 72.0% right after training. Mistakes fell 52%, from 695 to 333, and the score on rare categories doubled.
- Why it matters
- The same loop runs on your workflow. Your experts define correct, our verified reviewers build the answer key, and you keep the model.
- Cases tested
- 1,190
- Right before
- 41.6%
- Right after
- 72.0%
- Mistakes cut
- 52%
- Rare cases
- 2x
What we did, in plain English
Think of a mailroom.
Thousands of complaints arrive, and someone has to read each one and drop it in the right pile: which product, which problem. The wrong pile means it waits, bounces, or gets the wrong answer. We asked if a small AI model could learn to sort them the way the records say they belong.
- 001 Examples
Gave it real examples
5,514 real complaints from the CFPB public archive, each with the product and issue it was filed under.
- 002 Adapter
Taught it the job
We did not rebuild the model. We trained a small adapter that snaps onto it, like a lens. Take it off and the original is untouched.
- 003 Held out test
Tested it on new mail
1,190 complaints kept out of training entirely. Both versions got the same instructions and the same grading.
The numbers
Out of every 100 complaints.
Each square is one complaint. Green means it landed in the right pile.
58 of every 100 cases needed a person to fix them.
28 of every 100. The same model, now trained on the job.
Both the product and the issue had to match the record.
Every category counts the same, so the model cannot win by only learning the common ones.
Not luck.
We reshuffled the test 1,000 times and measured the gain again each time. In 95% of them the balanced score improved by +0.329 to +0.388, nowhere near zero.
And it did not win by guessing the popular answer. Always picking the most common category scores 0.018. The trained model scores 0.718.
What it could mean for you
Now picture your queue.
Uses the error rates from this test. It shows scale, not a promise: your real number comes from a pilot on your own cases.
Straight talk
What this proves, and what it does not yet.
What we proved
On real public data, training a small open model on the job cut its mistakes roughly in half, and the gain held up across 1,000 reshuffles. The whole loop ran end to end, from cloud training to testing on unseen cases.
What we have not proved yet
Nobody has independently rechecked the archive’s labels. This is not a customer result: it does not show lower cost or a better KPI at a real company. That is exactly what a pilot measures.
The details, for the curious
How the test was set up.
The setup, line by line
- Data
- CFPB public complaint narratives with their recorded product and issue labels.
- Split
- 5,514 to train, 1,180 to tune, 1,190 held out for the final test.
- Base model
- Qwen3-8B, pinned to one revision so anyone can repeat the run.
- Training
- One QLoRA pass, 690 steps, producing a separate adapter instead of changing the original weights.
- Scoring
- Exact match on product and issue, plus macro F1 so every category counts equally. Uncertainty from 1,000 paired group resamples.
- Speed
- 1.97s to 3.03s average per case with the adapter attached.
What is an adapter?
A small adapter that teaches a model one specialty. The original model stays exactly as it was, and the adapter can be attached or removed on its own.
What does held out mean?
Those cases were kept away from training entirely. It is how we check the model learned a real pattern instead of memorizing answers.
How sure are you about the labels?
The categories are the archive’s recorded labels, not an independently verified answer key. Human review is still needed before treating this as proof of correctness. Results are specific to this dataset, prompt, model revision, and setup.
Sources: CFPB public archive and the pinned Qwen3-8B revision.
Your turn
Swap the public labels for your own definition of correct.
Same test, on one workflow you already own. Your experts set the bar, our verified reviewers build the answer key, and the result is scored on a KPI you pick before training starts.
Goes straight to our CTO at rodrigo@joindrafted.com