Evaluations
Your agent regressed in prod. You didn't catch it.
Run a candidate revision against a golden dataset, score it with a judge you configure, and compare it to the revision currently deployed. Promote on evidence instead of on a vibe check.
Candidate versus deployed, on the same inputs
An evaluation run takes the revision you're considering and the one serving traffic, runs both against your dataset, and shows you the difference. The regression that matters is the one your users would have noticed — not one a synthetic test invented.
Deterministic runs, because integrations are sandboxed
Evaluation runs execute in sandbox mode, so every integration routes to its configured substitute — the vendor's test environment, the Fruxon simulator, or read-through. Your golden set can exercise Salesforce, Stripe, or your warehouse without polluting production or burning credits, and it produces the same result twice.
A gate, not a dashboard
Define criteria, let the judge score each result, and decide what clears the bar. Results live with the agent revision that produced them, and a failure links straight to the trace — so the fix loop is open the revision, change it, run it again.
How it fits together
Evaluations close the build loop.
A failing case points at a trace, the trace points at a revision, and the revision opens in Studio. You change the prompt or the tool wiring and re-run the same dataset — without leaving the agent or rebuilding the test.
Failures linked to the revision and the execution trace that produced them
Golden datasets are versioned alongside the agent, not stored in a spreadsheet
Same auth, same tools, same sandbox configuration as a real run
The rest of the build surface
Every capability is included on every plan — see pricing