Skip to content

Handing a verified tool surface to a benchmark

This project answers is this tool surface sound? It cannot answer will an agent use it correctly? — that is a measurement over agents, and tooltrace-bench is the sibling project that makes it.

apiverity agent-tasks is the hand-off.

apiverity agent-tasks tools.json -o packs/orders
tooltrace task validate --path packs/orders
tooltrace run --pack packs/orders --agent <yours>

What a generated task measures

For each tool in the manifest: given the contract, write the arguments for one call to it. The task is graded with the tool's own inputSchema — a json_schema assertion pointed at the schema this project just verified.

That is the whole idea. A tool manifest exists so that a caller can construct a valid call from it; whether an agent can is a fact about the agent, and it is measured against the contract rather than against somebody's opinion of the contract. A hand-written expected answer would be a rubric that can disagree with what the service actually publishes, and then the benchmark is measuring the rubric.

assertions:
  - type: file_exists
    params: {path: call.json}
    weight: 0.5
  - type: json_schema
    params:
      path: call.json
      schema:            # ← the manifest's own inputSchema, verbatim
        type: object
        required: [query]
        properties:
          query: {type: string, minLength: 1}
          limit: {type: integer, minimum: 1, maximum: 100}
    weight: 1.0

Every task carries a reference solution

tooltrace task test refuses a task with no metadata.scripted_script: without one, nobody has shown the task is solvable, and an unsolvable task in a benchmark is a score everyone loses for the author's reason. That objection is this project's own ethic arriving from the other direction, so it is honoured rather than worked around.

The solution is generated by apiverity.fuzz.generate.generate_valid — the same generator apiverity test uses — over the same schema the assertion grades against. Not a second implementation: a schema-to-example generator written for this export would be a parallel version of one this project already ships, and the two would disagree eventually.

tests/unit/test_agent_tasks.py then checks the reference solution against the assertion that grades it. That is a fidelity test on the contract model, not on the exporter: the solution comes from the SchemaNode model and the scorer is the raw JSON Schema from the manifest, so a constraint the model drops shows up as a reference answer that fails its own check.

Seeded from each tool's name, so re-running the export produces the same pack byte for byte.

Three things it refuses to do

Export a surface that did not pass. A pack built from a manifest this tool reports errors on benchmarks agents against a contract we say is broken, and every failure it reports is then ambiguous between the agent and the manifest.

$ apiverity agent-tasks tools.json -o packs/orders
  [ERROR] MCP-POISON-CREDENTIAL-PATH  tool 'get_weather' names '~/.ssh' …
error: 2 error-level finding(s) in this surface. Fix them, or pass
--allow-findings to export anyway -- the findings travel with the pack

--allow-findings overrides it and writes the findings into the pack's README, so the ambiguity is at least written down where the person reading the scores will see it.

Export a tool whose description was reported as poisoning. You cannot ask an agent to follow a description that is an injection and then score it for compliance: the correct behaviour is to refuse, and this task shape would mark that wrong. Measuring resistance is a different task shape, and pretending this one does it would be worse than not shipping it. --include-flagged exports them with the rule id in metadata.source.poisoning_rules, for an author writing that other shape by hand.

Export a task nothing can fail. A tool declaring {"type": "object", "properties": {}} constrains nothing: {} validates and so does anything else. A task whose assertion every submission passes is not a measurement, and a pack of them reports a score that means nothing. Those are skipped and the reason travels in the README — fixtures/mcp/tools_v1.json has one, list_regions, so this is exercised rather than hypothetical.

Why the export writes files instead of importing tooltrace-bench

An exporter that imported the sibling project would make it a dependency of this one and couple this project's releases to theirs. The output is YAML, held against tooltrace-bench's published schema, vendored under schemas/vendor/tooltrace-task-2026-09-11.schema.json the way the OAI's Arazzo schema is.

Published matters here. That project's working tree has tool_descriptions and attachments fields its published schema does not, and tool_descriptions would be the obvious way to measure resistance to a poisoned description. The task schema takes additionalProperties: true, so a pack using one would still validate — and the runner would ignore it. A test fails if either field is ever used, and fails the other way too, when the field becomes published and the decision is worth revisiting.

Verified end to end

The pack is not just schema-valid in theory. Against tooltrace-bench 0.3.0 on 2026-09-11, from fixtures/mcp/tools_v1.json:

$ tooltrace task validate --path pack
{'valid': 2, 'errors': []}
$ tooltrace task test --path pack
{'passed': 2, 'problems': []}

task test runs each reference solution through the scorers, so the second line is the loop closing: this project's generator, graded by this project's verified contract, inside the sibling project's runner.