Tier 3 · scoped build
Guardrail and evaluation platform
A golden set, an evaluation harness and a groundedness gate wired into your CI, so a release fails when accuracy degrades instead of a customer finding out.
Who this is for
A team already running AI in production that cannot currently tell whether today's release is better or worse than last week's.
From
$24,000
- Typical
- $40,000 to $80,000
- Other currencies
- €21,900 · £18,500 · S$32,000
Thirty minutes with the engineer who would run the work, before any paperwork.
The work, stage by stage
In order, with what happens at each phase and what it produces. Phases rather than dates, because a build is scoped per engagement and the catalog deliberately publishes a price band instead of a duration.
Golden-set construction
Phase 1
Cases from your real traffic and your real corpus, including the ones that have already gone wrong in production, each with the answer that should have been given.
Decide which metrics gate
Phase 1
Which properties block a release and which only report. A metric that cannot fail a build is a dashboard, and dashboards do not stop regressions.
Build the harness
Phase 2
Deterministic scoring where the property allows it, model-graded scoring where it does not, and the grader itself validated against human judgement on a sample rather than trusted.
Guardrails at runtime
Phase 3
Input and output checks for the failure modes the golden set exposed, placed where they can refuse rather than only record.
Wire it into CI and make it fail
Phase 4
The gate is proven by regressing the system deliberately and watching the build go red. An untested gate is an assumption, and this is the same discipline the harnesses on this site are held to.
Thresholds and handover
Phase 4
Thresholds set from a measured baseline rather than from round numbers, with the runbook for what to do when the gate fires.
Everything that arrives, listed
What is handed over at the end, and what you own of it. If something here would not be useful to you, say so before we start and the scope changes.
- The golden set, which is yours, with the provenance of each case recorded
- The evaluation harness, runnable locally and in CI
- The CI gate, proven by a deliberate regression that turned the build red, with that run recorded
- Runtime guardrails for the failure modes the golden set exposed
- Thresholds derived from a measured baseline, with the baseline published beside them
- A runbook for a firing gate, including how to tell a real regression from a flaky grader
What this covers, and who takes the rest
Stated up front, because scope is what derails an engagement in week two rather than at the proposal.
Inside this build
- Golden-set construction, the harness, the CI gate and the runtime guardrails
- Grader validation against human judgement on a sample
- Thresholds derived from a baseline measured on your system
Handled separately
Named so you can plan and budget for it, whether we do it or somebody else does.
- Fixing the model or retrieval failures the first baseline exposes, which is separate build work
- Running the suite on a cadence after handover, which the Maintain retainer covers
Start with a call, not a proposal.
Thirty minutes, straight to the engineer who would run the work. We will tell you if a smaller engagement answers your question, or if nothing here does.
Typical reply within one business day · Engagements start at $2,500
Scoped per engagement