Notes › 2026-11-04

What SWE-bench Verified tasks actually leak, task by task

At least 149 of the 500 SWE-bench Verified tasks give the fix away in the issue text or its comments; up to 163 if the tasks that state it on a second read are counted (checked 3 Nov 2026, methodology m1.1). Each flag is one quoted sentence from the task, shown in place, plus the date of the upstream fix commit. 131 of them are Tier A, where the text states the change; 18 are Tier C, where added lines of the gold patch reappear in the issue or comments (a plain string match, no model involved). Tier A precision in the labelled sample is 100% (39 of 39, 95% interval 91–100%).

Three of them to open and read yourself are in the next section. The whole table, filterable and downloadable as CSV, is at https://leaklens.grain64.com/benchmarks/swe-bench-verified?ref=note-1, and any task by instance ID, repo or PR number is a search away at https://leaklens.grain64.com/?ref=note-1. Running the same check on your own benchmark is free at https://leaklens.grain64.com/app?ref=note-1.

Three tasks to check by hand

1. django__django-11099: the issue proposes the exact regex.
The issue's own text says: “You can avoid this behavior by instead using \A and \Z to terminate regexes. For example, the validator regex could be changed to r'\A[\w.@+-]+\Z' in order to reject usernames that end with a newline.” It then names the two validators to change. The fix commit is dated 2019-03-22 and has been public since.
Task page: https://leaklens.grain64.com/t/django__django-11099?ref=note-1

2. django__django-10914: the title is the fix.
The issue is titled “Set default FILE_UPLOAD_PERMISSION to 0o644.” The upstream commit is dated 2019-02-08.
Task page: https://leaklens.grain64.com/t/django__django-10914?ref=note-1

3. astropy__astropy-14508: a maintainer's comment names the change.
In the discussion: “So I think replacing this by value_str = str(value) should be totally fine.” Fix commit dated 2023-03-13.
Task page: https://leaklens.grain64.com/t/astropy__astropy-14508?ref=note-1

That is a different claim from "models were trained on these tasks". It is narrower and checkable: a model that sees the prompt sees the answer, whatever it has or has not seen before. For the commit side, the right wording is public exposure possible: the fix has been public since a date you can read next to the flag.

The headline and the range

• At least 149. The flagged count is the tasks where two model passes, with differently worded instructions, reach the same tier and the quoted sentence appears verbatim in the text.
• Up to 163. 60 tasks are disputed (the two passes differ). A second model read of those 60 found 14 that state the fix, 13 that point to it and 33 with neither. Those 14 stay outside the 149 and outside the per-tier precision; counted, the total is 163.
• 69 more were settled as Tier A by a third check and are counted outside the 149; their sampled precision is 80% (16 of 20, 95% interval 58–92%).
• Tiers. Tier A: the text states the fix. Tier B: it names the exact file, function or line without giving the change. Tier C: added lines of the gold patch also appear in the text.
• Commit dates are shown, not scored. Every SWE-bench task is built from a merged upstream pull request, so its fix is in public history by construction. The date tells you when, which matters if you care about a given model's cutoff.

Full definitions, versions and limits: https://leaklens.grain64.com/methodology?ref=note-1

Why this matters now

• SWE-Bench+ manually screened the patches that SWE-Agent + GPT-4 resolved on SWE-bench and found the solution given in the issue report or comments for 32.67% of them; after filtering those and weak-test instances, the resolve rate fell from 12.47% to 3.97% (paper).
• OpenAI stopped reporting SWE-bench Verified after an audit of 138 tasks that models consistently failed, 59.4% of which it found to have flawed tests (OpenAI, Decrypt).
• A probing study found models can name the file to edit from the issue text alone far more often on Verified (up to 76%) than on repositories outside SWE-bench (up to 53%) (SWE-Bench Illusion).

Those are verdicts about the benchmark. What was missing is the list, task by task, with the sentence. That is the table. SWE-bench Lite is checked too (300 tasks, 72 Tier A and 12 Tier C): https://leaklens.grain64.com/benchmarks/swe-bench-lite?ref=note-1.

How far to trust a flag

The figures come from labelled samples, and the full method, seed and every disagreement are listed at https://leaklens.grain64.com/validation?ref=note-1. In short: Tier A held in the sample (39 of 39); Tier C held in 17 of 18, the one false positive being sympy__sympy-19637, where a repeated line is the failing line of a traceback, not the fix. Among unflagged tasks read, 4 of 47 (9%, 95% interval 3–20%) were found to leak against the gold patch, which is why 149 is a floor. If you find a flag that is wrong, report it at https://leaklens.grain64.com/contact?ref=note-1 and the correction goes in the dated log at https://leaklens.grain64.com/new?ref=note-1. One more benchmark or split is checked each week and logged on the same page.

Cite the table

Leaklens, SWE-bench Verified, methodology m1.1, 3 Nov 2026: 149 of 500 tasks flagged (131 Tier A and 18 Tier C, where both checks agreed); Tier A precision 100% (39 of 39, 95% interval 91–100%); Tier C precision 94% (17 of 18, 95% interval 74–99%); 69 more settled as Tier A by a third check, counted outside the 149, precision 80% (16 of 20, 95% interval 58–92%); 60 disputed (12%); miss rate 9% (4 of 47, 95% interval 3–20%) among unflagged tasks read; at least 149 flagged, and up to 163 if the 14 disputed tasks that state the fix on a second read are counted; a second model read (same model family, blind extraction) of those 60 disputed tasks found 14 that state the fix, 13 that point to it and 33 with neither (tiers unchanged). https://leaklens.grain64.com/benchmarks/swe-bench-verified?ref=note-1

@misc{leaklens_what_verified_tasks_leak,
  title = {What SWE-bench Verified tasks actually leak, task by task},
  author = {{Leaklens}},
  year = {2026},
  url = {https://leaklens.grain64.com/notes/what-verified-tasks-leak},
  note = {Table: https://leaklens.grain64.com/benchmarks/swe-bench-verified, methodology m1.1}
}