We write down what correct means for one job your team does, then make every model and vendor take that test on your real cases. You see who passes, own the winner, and retest it on every update.
Charged twice for one purchase and the bank will not reverse it.
Answer keyCredit card / Billing dispute
Not graded yet.
Each candidate graded against the answer key, case by case
Candidate
Score
Frontier API APaid per call
0/8
Frontier API BPaid per call
0/8
Open modelOff the shelf
0/8
Open model + your keyRuns in your cloud
0/8
Grading against the key…
An illustration of how a test reads. Your test uses your cases.
Every demo looks good. None of them ran on your work.
Most enterprise AI stalls between pilot and production for one reason: nobody can show, with evidence, that it is good enough for this job. A test built from your own cases settles that, and it keeps settling it every time the model changes.
Four steps, end to end.
Start with the first. Each step stands on its own, and each one makes the next one cheaper.
01Acceptance TestWeeks 1 to 3
Write down what correct means
What does a right answer look like here?
Your experts set the rules for one job. Our verified reviewers grade your real cases against them, and your team signs off. Correct stops being an opinion.
“My mortgage payment went up and nobody explained the escrow change.”
Reviewer 1Mortgage / Escrow
Reviewer 2Mortgage / Escrow
Reviewer 3Mortgage / Payment trouble
Adjudicated and signed off by your expert: Mortgage / Escrow
You get
An answer key built from your own cases
A rubric your team approved
Agreement scores, so you know the key itself holds up
Under the hood
Sampling
Cases drawn across every category, rare ones included, so a model cannot pass by learning only the common ones.
Grading
Each case graded independently by several identity verified reviewers. Disagreements go to an adjudicator, then to your expert.
Agreement
Inter rater agreement reported per category. Categories that do not reach the bar get a clearer rule, not a coin flip.
Privacy
Sensitive fields redacted on ingest. The key lives in your environment.
02Acceptance TestWeeks 3 to 5
Make every option take the test
Which model or vendor is actually good at this?
Frontier APIs, open models, a vendor’s agent. Same cases, same instructions, same grading. You see who passes before you sign anything.
Your bar
Frontier API A
6/8
Frontier API B
5/8
Open model
3/8
Open model + your key
6/8
You get
A scorecard for every candidate, against a bar set in advance
Accuracy on rare cases, cost per thousand cases, and speed
The failures themselves, with examples, not just a number
Under the hood
Held out
Test cases are never shown to any vendor or used in training, so nobody can tune to the test.
Scoring
Exact match where the answer is a label, a rubric where it is open ended.
AI judges
Used only after we calibrate them against human grades, and we publish how often they agree.
Confidence
Every score comes with a resampled interval, so a lucky run cannot pass as a real win.
03Own the ModelAbout 3 weeks after the test
Own the winner
Can we get the same result cheaper, and keep it?
If a small open model can be trained to pass, we train it on your answer key and hand it over. It runs in your cloud, with no per call fees and no data leaving.
41.6%72.0%
Our public test: the same open Qwen3-8B model, before and after training, on 1,190 cases it never saw. Read the case study
You get
A trained model running in your cloud, yours to keep
Before and after scores on the same unseen test
A handoff runbook your engineers can operate
Under the hood
Method
A small adapter trained on open weights pinned to one revision. The base model is never changed.
Data
Trained only on the training split of your key. The test split stays untouched.
Deploy
Your VPC, on premises, or an inference provider you choose.
04StewardshipMonthly
Keep it passing
Is it still good after the next update?
Models change every few weeks and so does your work. We rerun your test on every release and vendor update, and grow the key as new edge cases show up.
Example: release 5 slipped under the bar and was caught on the retest, before it reached your queue.
You get
A fresh scorecard on every model release
An alert when a score drops below your bar
An answer key that grows with your real traffic
Under the hood
Versioning
Every key and every score is versioned, so any result can be reproduced later.
Refresh
New cases sampled from production each cycle and graded by the same reviewer cohorts.
Audit
A human audit of a slice every cycle keeps automated judges honest.
How you buy it.
Three products, one answer key underneath all of them.
Acceptance TestStart here
Which option is good enough?
Your answer key and a scorecard on every candidate you are weighing
About 5 weeksFixed price, one workflow
Own the Model
Can we own one that is?
A trained open model in your cloud, scored on the same test
About 3 weeksFixed price
Stewardship
Is it still good?
A retest on every release, drift alerts, a key that keeps growing
OngoingMonthly retainer
Sketch your test in a minute.
Pick the job and who should take the test. We reply with a fixed scope and price.
Your test plan
The test
Real ticket routing cases, graded into an answer key your team signs off.
Who takes it
Frontier APIs and Open models
Scored on
Accuracy on common and rare cases, against a pass bar agreed before anything runs.
You keep
The answer key, the scorecard, and the failure examples, whoever wins.
Acceptance Test
Own the Model
Stewardship
Email this planOpens your email with the plan filled in. Goes to our CTO.
Independent, in writing.
Your data trains nothing else.
Not our models, not a lab’s. Your cases and your key stay yours, and the contract says so.
No vendor pays us.
We do not resell models or take referral fees, so the scorecard has no reason to favor anyone.
Bad news travels first.
If nothing passes your bar, including the model we trained, you hear it from us before anyone else.
The questions your team will ask.
Who are the reviewers?
Students enrolled at partner universities, identity checked through their institution. The same cohorts return each semester, so the quality of your key does not reset. For licensed work like clinical or legal judgment, your experts stay the final authority.
Why not just grade with an AI judge?
You can, once you know you can trust it. We calibrate any automated judge against human grades on your cases and report how often they agree. Where it agrees, it scales the test cheaply. Where it does not, people grade.
Can closed models take the test?
Yes. Any model or agent you can call can take it. Owning the winner only applies to open models, because a closed model cannot run in your cloud.
Where does our data live?
In your environment. Sensitive fields are redacted on ingest, reviewers work only inside the access you grant, and nothing is shared with model providers beyond the calls you approve.
What does it cost?
The Acceptance Test is a fixed price for one workflow, sized on a short call once we know the volume and how many candidates you are weighing. Owning the model is a fixed add on. Stewardship is a monthly retainer.
Before you sign that contract, we build the test.
Bring one workflow and the options you are weighing. We will tell you if it is worth testing, including when it is not.
