Notes › 2026-11-04

Pre-release leakage checks for a new coding benchmark

Before you publish a coding benchmark, read your own tasks the way a model sees them: the prompt, the comments, and the commit history that already exists. Four checks cover the leak types reviewers have already found in older benchmarks; each can be run on the public tasks today, and each leaves evidence you can publish with the release.

Why bother: the leaks that sank trust in older benchmarks were not subtle. SWE-Bench+ manually screened the patches that SWE-Agent + GPT-4 resolved on SWE-bench and found the solution given in the issue report or comments for 32.67% of them; filtering those and weak-test instances moved that agent's resolve rate from 12.47% to 3.97% (paper). OpenAI stopped reporting SWE-bench Verified after its audit (OpenAI). New releases (SWE-bench Pro publishes 731 tasks, holds out 858 and keeps 276 commercial ones back — Scale) are read against that history.

The four checks

1. Does the issue state the fix?
Search each task's issue and comments for a sentence that gives the change: a code snippet, "just change X to Y", a maintainer comment naming the line. Quote it and keep the link. On SWE-bench Verified, at least 149 of 500 tasks give the fix away in the text (131 of them state it outright, with two model passes agreeing, and up to 163 are flagged on a second read); one is django__django-11099, where the issue itself spells out the replacement regex (task page). The same check on SWE-bench Lite, released earlier and widely used, states the fix outright in 72 of its 300 tasks (Lite table). Both numbers are floors: the unflagged side of Verified still hides leaking tasks: 4 of 47 unflagged tasks read (9%, 95% interval 3–20%) in a labelled sample (validation).

2. Does the issue point at the exact place without giving the change?
File, function or line named is weaker evidence than a stated fix, but it narrows the search to almost nothing. Probing studies show models can often identify the edited file from the issue text alone (SWE-Bench Illusion). Count these separately from stated fixes.

3. Do lines of the gold patch appear in the issue or comments?
Diff the patch against the text. Reproduced lines are a mechanical signal that does not depend on anyone's reading. Beware the false positive: a line in a traceback is not a fix (Leaklens' own validation lists one such sample, sympy__sympy-19637, on the validation page).

4. When did the fix become public, and where else did it go?
Every task that comes from a merged upstream pull request has its fix in public history. What changes between tasks is the date, relative to the training cutoffs you care about. Record the commit date and link beside each task. Word it as public exposure possible: a date shows the fix was available, not that any model saw it. Forks and mirrors of the upstream repository extend the exposure; note them too.

What to publish with the release

• A per-task table: instance, tier, the quoted sentence, the commit date and link.
• The method, with a version number, and how to reproduce a flag by hand.
• A labelled sample of your flags and of your unflagged tasks, so readers see both error directions.
• A correction path: where to report a wrong flag and where corrections are logged.

A flag a reviewer can check themselves is what a release needs.

Get your public tasks checked

By hand, you read every task's issue, comments and patch yourself. Leaklens runs these four checks and shows the evidence on public pages. Worked examples, each with every flag quoted and linked: SWE-bench Verified at https://leaklens.grain64.com/benchmarks/swe-bench-verified?ref=note-2 and SWE-bench Lite at https://leaklens.grain64.com/benchmarks/swe-bench-lite?ref=note-2. The method is at https://leaklens.grain64.com/methodology?ref=note-2.

Send a public repo + commit list (CSV: repo,base_commit,instance_id) or a split URL. The per-task report is sent back: tier, the quoted sentence, the commit date and link for every task. Free during early access. Request it at https://leaklens.grain64.com/contact?ref=note-2 and choose “Audit my benchmark”.

@misc{leaklens_pre_release_leakage_checks,
  title = {Pre-release leakage checks for a new coding benchmark},
  author = {{Leaklens}},
  year = {2026},
  url = {https://leaklens.grain64.com/notes/pre-release-leakage-checks},
  note = {Table: https://leaklens.grain64.com/benchmarks/swe-bench-verified, methodology m1.1}
}