Evaluations
Behavioural fixtures asserting exact tool-call arguments, run before a deploy.
A fixture states the input and the tool call that must result, with its exact arguments. It does not compare output text, because output similarity passes while the agent calls the wrong tool with plausible arguments — which is the failure that matters.
The fixtures run in continuous integration before a deploy, so an agent that stops calling the right tool fails the build rather than degrading quietly once deployed.
- Behavioural fixtures assert exact tool-call arguments, not output similarity, and they run in continuous integration before a deploy.
- A conversational agent that stops calling the right tool fails the build rather than degrading quietly once deployed.
- Per-conversation model cost is reported to a budget surface, so an agent that becomes expensive is visible as a number.
How it works
Cost is reported per conversation against a budget surface, which makes an agent that has become expensive visible as a number rather than as a surprise on an invoice. The budget it reports against is the per-environment one in Environment.