Methodology

Bradley–Terry calculation

P(A beats B) = 1 / (1 + 10^((Elo_B − Elo_A) / 400))
  • Mean 1000; Gaussian prior SD 400 Elo.
  • Ties count as ½ win.
  • Table intervals are 95% posterior credible intervals.

Excluded data

  • Internal testers.
  • Battles where either contestant failed.
  • Battles where either contestant has $0 recorded account spending, including sponsored provider endpoints.
  • Incomplete records.
  • Battles affected by the historical subagent bug.
  • With Exclude early voting: early picks and battles stopped before both contestants finished.

Bootstrap sample calculation

  • Developer-cluster bootstrap; B = 1,000 replicates.
  • Each replicate draws D developers uniformly with replacement from D observed developers, retaining all their battles and multiplicity.
  • Refit the maximum a posteriori Bradley–Terry estimate for each replicate.
  • Tied rank positions split their weight equally.
  • Ridgeline smoothing: Gaussian kernel density estimates, with bandwidth h = max(2, 1.06sB−1/5) Elo; s is the replicate standard deviation.
  • Ridgeline axis: from the lowest 0.5th to the highest 99.5th percentile of the ranked models' curves, rounded out to 25 Elo.

Battle-adjusted cost and time

Reported cost and time are adjusted for differences in the average difficulty of the battles each model received, while retaining each model's observed variation and long-running or expensive cases.

log(value_A / value_B) = model_effect_A − model_effect_B + error
  • Cost and time are fitted separately by least squares of within-battle log ratios. Each battle has equal weight. A shared multiplicative task effect cancels in each comparison; model-specific task effects may remain.
  • Both contestants must have completed, without an early vote or a stop while running. Each metric requires finite, positive measurements on both sides. Missing and zero measurements are not imputed.
  • Historical opponents connect the fit before the current model pool is selected. Claude Opus 5.5 anchors the connected graph; estimates require at least 20 paired battles for both the model and the reference. Disconnected or insufficiently sampled models have no estimate.
  • Each battle's task effect is the average of its contestants' log values minus their fitted model effects. For each model, we compare the average task effect in its own battles with the average across all connected battles. Its adjustment factor is exp(pooled average task effect − its own average task effect). Harder-than-average assignments receive a factor below one; easier assignments receive a factor above one.
  • We multiply that model's recorded observations by its adjustment factor, then report their median and arithmetic mean in dollars or minutes. Equivalently, each adjusted summary is its raw summary times that model's factor. This preserves its mean-to-median ratio and observed distribution shape. Models that faced exactly the same battles receive no adjustment. The main table uses the median; the chart and model details also offer the mean.
  • This is an adjustment for average difficulty on the log scale, not a reconstruction of every model on an identical workload. Differences in the spread of task difficulty, model-specific task effects, and changes over time can remain. Changing the numerical reference leaves the correction unchanged within the same connected graph.
  • Raw and adjusted cost/time rows in model details use the same completed, non-early paired observations, so their difference reflects only the adjustment. They stay fixed when the early-vote control changes Elo. Older snapshots without adjustment data retain their original raw summaries until refreshed.

Provider selection

  • OpenRouter routes each model through a model-specific provider and endpoint allowlist.
  • Routing prioritizes throughput, requires support for the requested parameters, and applies per-model price caps.

Definitions

  • Rank likelihood: share of bootstrap samples at each rank.
  • Ridgelines: smoothed Elo distributions from bootstrap samples.
  • Raw win rate: excludes ties from wins.
  • Tokens per second: median per-run token rate.
  • Cost / time: battle-adjusted median or arithmetic mean per task, in dollars and minutes.
  • Pareto frontier: ranked models with no other ranked model having equal or higher Elo and equal or lower selected cost or time, with at least one strict improvement.
  • Pareto bars: 95% Elo confidence intervals.
  • Thin / thick bars: thin bar is the 95% confidence interval; thick bar is the 50% confidence interval.