01 / The Brief / 30%
Read the messages, spreadsheet and three summaries. Pick one and defend the decision in 80 words. The existing scorer grades the choice; the panel grades the evidence and workspace transcript.
The model benchmark
Can the model do the work?
The full laptop assessment, translated into three versioned eval sets. Models judge a summary, audit a recommendation, and build a workflow that must hold up on unseen cases.
Model performance
Same tasks. Models in the hot seat.
01 / 30%
The Brief
02 / 30%
The Catch
03 / 40%
The System
Loading measured results…
Version … · 3 tasks · 10 held-out cases
What each eval measures
02 / The Catch / 30%
Review all five memo sections against their sources. Flag spans, quote proof, propose fixes, and decide send or hold. Existing checks score defect recall, precision and the 20,000-token budget.
03 / The System / 40%
Build and test a valid staged workflow on five visible cases. The existing runner executes ten held-out cases. The report shows unseen-case accuracy, the visible-to-held-out gap, routing and measured cost.
How to read the results
The model acts as the candidate, with the existing workspace model tiers available as tools. This is an automated translation of the computer test. It does not measure mouse use, typing, or the human rule against pasting.
Each round keeps its 7 / 8 / 10-minute ceiling and allows up to eight actions. Round 2 counts workspace tool tokens, as in the human test. The candidate model’s own inference is recorded in total benchmark cost, outside that allowance. Models have two visible workflow runs to inspect outputs and revise; held-out inputs, labels and grading answers never enter the candidate context. All models use the same tool menu and prompts.
The board reports means and completed-trial counts for the current eval version, with failures shown separately. A single run is an observation, not a reliable ranking. The grading panel is held constant across models; some judge and candidate pins may share a model family. Model submissions do not receive human credentials or enter the human leaderboard.
Only full runs with an available grading panel and ten held-out case records receive scores. Infrastructure or budget failures receive no ranking. The eval version covers task inputs, grading data, rubric and protocol; each private run artifact also records the source commit, model pins, judge pins, submitted work and grading evidence. The tasks are public, so this is not a claim of an uncontaminated research benchmark.
Read the assessment rubric ↗