Notes

What the checked tasks actually show, written so you can verify each claim by hand.

• Pre-release leakage checks for a new coding benchmark
  2026-11-04 · Four checks to run on a coding benchmark's public tasks before release, what to publish with it, and a worked example on SWE-bench Verified (at least 149 of 500 tasks flagged) and Lite.
• What SWE-bench Verified tasks actually leak, task by task
  2026-11-04 · At least 149 of the 500 SWE-bench Verified tasks give the fix away in the issue or its comments, up to 163 on a second read. Each flag quotes the sentence and dates the upstream fix. Three tasks to check by hand.