Notes › 2026-11-12

SWE-bench Verified leaderboard entries, rescored without the flagged tasks

Take the 135 SWE-bench Verified submissions with a published results file, remove the 149 tasks whose issue or comments state the fix, and the best entries still solve about 74% of what is left, down from 79% reported. Rescored 12 Nov 2026 from the per-task resolved lists in the public SWE-bench experiments repository, on Leaklens methodology m1.1. Arithmetic only: no model was run.

Every entry, with its reported score, its score on the flagged tasks and its score on the rest, is one file: https://leaklens.grain64.com/swe-bench-verified-rescored.csv?ref=rescore . Find your own submission by its folder name.

The ten highest reported entries

• 20251205_sonar-foundation-agent_claude-opus-4-5: 79.2% reported (396 of 500), 74.1% on the 351 unflagged tasks (260), 91.3% on the 149 flagged (136)
• 20251215_livesweagent_claude-opus-4-5: 79.2% reported (396 of 500), 73.8% on the 351 unflagged tasks (259), 91.9% on the 149 flagged (137)
• 20250928_trae_doubao_seed_code: 78.8% reported (394 of 500), 74.1% on the 351 unflagged tasks (260), 89.9% on the 149 flagged (134)
• 20251127_openhands_claude-opus-4-5: 77.6% reported (388 of 500), 72.4% on the 351 unflagged tasks (254), 89.9% on the 149 flagged (134)
• 20251120_livesweagent_gemini-3-pro-preview: 77.4% reported (387 of 500), 72.9% on the 351 unflagged tasks (256), 87.9% on the 149 flagged (131)
• 20250804_epam-ai-run-claude-4-sonnet: 76.8% reported (384 of 500), 71.8% on the 351 unflagged tasks (252), 88.6% on the 149 flagged (132)
• 20250902_atlassian-rovo-dev: 76.8% reported (384 of 500), 71.8% on the 351 unflagged tasks (252), 88.6% on the 149 flagged (132)
• 20250819_ACoder: 76.4% reported (382 of 500), 71.2% on the 351 unflagged tasks (250), 88.6% on the 149 flagged (132)
• 20250901_warp: 75.6% reported (378 of 500), 71.5% on the 351 unflagged tasks (251), 85.2% on the 149 flagged (127)
• 20250612_trae: 75.2% reported (376 of 500), 70.7% on the 351 unflagged tasks (248), 85.9% on the 149 flagged (128)

What the numbers say

• About 5 points. Across the 20 highest reported entries the score on the unflagged tasks is 4.9 points lower on average than the reported score.
• The top 10 stays the top 10. Ranked by tasks solved, then by tasks solved among the unflagged 351, the same ten entries lead both lists.
• Positions inside the list move. 90 of the 135 entries change rank, the largest moves being 7 places (20250710_bloop and 20250520_openhands_devstral_small fall 7; 20241029_epam-ai-run-claude-3-5-sonnet rises 7).
• Flagged tasks are easier for everyone. The median entry solves 19.9 points more of the flagged tasks than of the rest, and so do the 2023 retrieval baselines: 20231010_rag_claude2 solves 6.7% of the flagged and 3.4% of the unflagged. A gap on its own is therefore not evidence that a given submission saw a task before. The rescored column answers a narrower question: how much of this number rests on tasks whose own text states the fix.

Method, and what to check by hand

• The flagged set is the 149 tasks of the Leaklens headline count: Tier A where two checks agreed (131) and Tier C (18). A stricter column removes 69 more Tier A tasks that a third check settled (218 in all, 282 left).
• A task is flagged when a quoted sentence from its issue or comments states the fix. Open any task page to read the sentence: for example https://leaklens.grain64.com/t/django__django-11099?ref=rescore and https://leaklens.grain64.com/t/django__django-10914?ref=rescore .
• Coverage: the repository's evaluation/verified folder holds 182 entries; 135 have a results/results.json at the standard path and are included. The other 47 are not scored here.
• Resolved lists are taken as published by each submitter. They were not rerun.
• Reproduce it: the CSV gives the counts per entry; the flag list is https://leaklens.grain64.com/benchmarks/swe-bench-verified?ref=rescore , downloadable as CSV.

How far to trust a flag

The figures come from labelled samples; the method and every disagreement are at https://leaklens.grain64.com/validation?ref=rescore . Tier A held in the sample (39 of 39); Tier C held in 17 of 18. Entry owners can report a correction on any task page and the page will say what changed.

Cite the table

Leaklens, SWE-bench Verified, methodology m1.1, 8 Nov 2026: 149 of 500 tasks flagged (131 Tier A and 18 Tier C, where both checks agreed); Tier A precision 100% (39 of 39, 95% interval 91–100%); Tier C precision 94% (17 of 18, 95% interval 74–99%). Leaderboard rescoring: https://leaklens.grain64.com/swe-bench-verified-rescored.csv?ref=rescore

@misc{leaklens_verified_leaderboard_without_leaked_tasks,
  title = {SWE-bench Verified leaderboard entries, rescored without the flagged tasks},
  author = {{Leaklens}},
  year = {2026},
  url = {https://leaklens.grain64.com/notes/verified-leaderboard-without-leaked-tasks},
  note = {Table: https://leaklens.grain64.com/benchmarks/swe-bench-verified, methodology m1.1}
}