Evals / QA · retainer
We put numbers on what your AI gets wrong
Test sets built from your real cases, accuracy metrics you can quote, and regression checks on every release. It doesn't matter who built the system.
How we measure the quality of your AI system
We agree what a correct result means, measure the current system on real cases, and rerun the same checks after each change.
Quality beyond the demo
We measure the system on real inputs, separately from the cases used for a demonstration.
Regression checks for each change
A change can improve one answer while reducing quality elsewhere. Regression checks measure the whole agreed test set before a release reaches users.
Numbers for the people who decide
An agreed metric gives budget owners and incident reviewers a measured result instead of an impression.
What you get
Quality control that keeps running. A one-off report is already stale by the next release.
- Test sets built on your data
- Real cases pulled from your own logs and users.
- Accuracy metrics you can quote
- An agreed definition of “correct”, then measured and reported as numbers you can take to stakeholders.
- Regression checks on every change
- New prompt, new model, new data. The eval suite runs before any of it reaches users, so a fix to one answer stops quietly breaking others.
- A growing eval suite
- Every production incident becomes a test case. The suite gets stricter as your system ages.
How it works
Collect real cases
The questions that come up, taken from your logs, your support tickets and your users.
Agree the metrics
Together we define what “correct” means for your system and set the bar to clear.
Baseline current quality
The first measurement of where the system stands today, on cases you chose rather than we did.
Guard every release
Regression runs on each change and reports over time. That's the retainer part.
We hold our own work to this standard
Projects we shipped behind eval pipelines, plus open results anyone can check.
In production
Expert Blockchain Chatbot
Production RAG with quality we measured. Retrieval goes past plain vectors (SQL, entity lookup, real-time data), and an evaluation framework (RAGAS) scores the answers.
Read the project →In production
LetAI: Nutrition Estimation Agent in Production Chat
The agent only reached production chat after its evaluation pipeline was expanded, with new datasets and broader case coverage added before release.
Read the project →In production
Market Analyst for a Chemical Manufacturer
Recommends what to produce next on evidence a buyer can re-check: every figure traced to its source row and cleared by named quality gates.
Read the project →Open code
ARLC 2026 Legal RAG Challenge
3rd of 155 teams in the challenge's warm-up phase, scoring 0.954/1.0. An agentic RAG pipeline that answers questions over real DIFC legal documents, with every answer graded by AI judges. The code is open, so anyone can check it.
View the code ↗Open code
BitGN Personal Agent Challenge
We built an AI agent for a safety challenge that hides prompt-injection attacks inside ordinary tasks. The agent has to get the work done and turn down the traps, and it's scored deterministically by what it actually changed. Our defense layer checks 40+ attack patterns across 14 categories. The code's on GitHub.
View the code ↗Publication
zebra_simple: Zebra Puzzle Test for LLMs
Our published LLM reasoning benchmark. Open evidence of how we measure model behaviour, and anyone can check it.
Read the publication →
Frequently asked questions
An eval suite checks the full agreed set on every release and gives you a measurement for comparison. Manual review remains useful for cases that need human judgement.
Evals are vendor-independent. We measure how the system behaves from the outside, on your data, whoever built it.
No. Evals show the current error rate, whether it is improving, and whether a change introduces a regression before release. They make mistakes visible and measurable; they do not make them impossible.
Unit tests check for an exact output, and language models don't give you exact outputs. Evals score behaviour across a whole dataset (accuracy, groundedness, whether it refuses when it should) and track those scores over time, which is what you need for a system that isn't deterministic.
Eval runs on every change, regular quality reports, and growing the test sets as new cases show up in production. Price depends on your system and how often you ship, so we quote after a scoping call. The form below starts one.
No AI in production yet? The two-week agent build starts with the task that pays for itself first, and ships it.
Vendor-independent
Discuss your case
Tell us what your AI does and what's worrying you about it.