Agents write Opentrons OT-2 protocols for 11 wet-lab tasks: 7 easy tasks that give the steps, 4 hard tasks that
give only a goal and the source paper (paper-only). Each protocol is simulated, checked against a ground truth, and scored by a
three-vote rubric judge. One attempt per task, same Claude Code agent for every model. Run 2026-10-07-openrouter.
Refused tasks are not scored: a refusal is the provider's safety policy, not a protocol. Models are ranked on the 8 tasks every model answered; refusals are listed beside the score.
Hover a score for the judge votes and any failed check; † marks a provisional score. Every failure with its evidence is in the full report.
| Task | Claude Sonnet 5.5 | Claude Opus 5.5 | Claude Fable 5.1 | GPT-6.1 Sol |
|---|---|---|---|---|
| Easy · steps given | ||||
a1-a12-100ul | 1.00 | 1.00 | 1.00 | 1.00 |
ampure-bead-cleanup | 1.00 | 1.00 | 1.00 | 1.00 |
colony-pcr-screening | 1.00 | 1.00 | 1.00 | 1.00 |
ecoli-heat-shock-transformation | 1.00 | 1.00 | 1.00 | 1.00 |
golden-gate-assembly | 1.00 | refused | refused | 1.00 |
opentrons-rna-extraction | 0.30 | 1.00 | 0.75 | 1.00 |
split-200ul-two-wells | 1.00 | 1.00 | 1.00 | 1.00 |
| Paper-only (hard) · goal and paper | ||||
colony-pcr-screening-hard | 0.75† | 1.00 | 1.00 | 0.50† |
ecoli-heat-shock-transformation-hard | 0.75† | 0.75 | refused | 0.75 |
golden-gate-assembly-hard | 1.00 | refused | refused | 0.75 |
opentrons-rna-extraction-hard | 0.69† | 1.00 | 1.00 | 0.69† |