OPEN METHODOLOGY / VERSION 0.1
How to measure action discovery for AI agents.
This page defines the reproducible design used by the first public Working Machines discovery snapshot. Results remain separate from the method and include prompts, ranked action IDs, manual gold sets, raw JSON, and limitations.
VIEW FIRST DATASET →Research question
When an agent receives an outcome-oriented task, how effectively can an application layer identify the smallest sufficient set of actions without placing an entire integration catalog into model context? The benchmark separates discovery, authorization, execution readiness, and verification so a system cannot score well merely by returning a long list of tools.
Required test controls
- Fixed model and model version
- Fixed system and user prompts
- Recorded catalog snapshot
- Fresh session for every task
- Identical connection availability
- Warm and cold latency reported separately
- Raw discovery response retained
- Human-reviewed gold action set
Metrics
Discovery recall
Did the system surface every action required by the task?
Selection precision
What share of surfaced actions were actually relevant?
Schema context cost
How many tokens or bytes were loaded before selection?
Time to executable plan
How long until the agent had valid action names and required inputs?
Authorization clarity
Could the agent distinguish missing access from a missing capability?
Verification completeness
Did the result contain enough identity and status data to prove what happened?
Task-set design
Use at least three task classes: single-app retrieval, single-app mutation, and multi-app workflow. Every task must state the expected actions, acceptable alternatives, forbidden actions, required confirmation points, and verification criteria. Include ambiguous tasks where the correct behavior is to ask for clarification rather than execute.
Publication requirements
- 1. Publish task prompts and gold action sets.
- 2. Publish model, client, catalog, and runtime versions.
- 3. Publish raw outputs with secrets and personal data removed.
- 4. Report failures, excluded runs, retries, and confidence intervals.
- 5. Keep commercial conclusions separate from measured results.
- 6. Add a dated changelog whenever the benchmark changes.