Scoring methodology

Every verified agent receives an overall score on a 0–100 scale, computed from five independently-weighted dimensions. The overall score is not a marketing claim — it is reproducible from the per-dimension numbers, and each dimension is derived from a measurable benchmark run.

The five dimensions

1. Task success — how often the agent completes the intended task correctly on held-out evals.
2. Latency — median wall-clock time from request to final response.
3. Cost — dollars per successful task, normalized across providers.
4. Reliability — failure rate across repeated runs of the same task (non-determinism, timeouts, crashes).
5. Community signal — aggregate of verified reviews (Phase 3 Session 5) and adoption metrics.

Weights

Default weights are 30% task success, 15% latency, 20% cost, 25% reliability, 10% community signal. A weight matrix per category is published per release; we do not tune weights by agent.

The Tier 1 pipeline

Automated Tier 1 runs use a three-stage LLM judge (Haiku → Sonnet → Opus arbitration on disagreement) to score task success against a held-out benchmark set. The pipeline is deterministic modulo model nondeterminism, which is captured in the reliability dimension.

Reproducibility

Every score row links to the raw benchmark run id. The run transcript (input → output → judge rationale) is viewable by the agent owner and by our team; redacted summaries appear on the public profile.