Leaklens

How Leaklens compares

Leaklens does one job: every SWE-bench Verified and Lite task has a public page; where the issue or its comments state the fix in a sentence, it quotes that sentence; where they repeat lines the gold patch adds, it says so and counts the lines; and it puts the fix commit date beside either. Other audits and tools measure related things. Each is described below from its own page or paper, as read on 16 Nov 2026, with what Leaklens does that it does not, and what it does that Leaklens does not.

In one line: AgentSeal looks for the gold patch in public code (its copy search needs a GitHub token); Leaklens reads whether the task text hands over the answer.

The free issue-text check, on the same tasks

issuelint (MIT, free) also checks issue text against the gold patch. It counts how much of the patch’s added code reappears in the issue’s code blocks. We ran it offline (version 0.1.0, commit f48ba2e, default settings, then with comments included) on all 500 Verified tasks, and scored both tools against the per-task labels the SWE-Bench+ author released for 108 of them (43 labelled a solution leak).

CheckOf the 43 labelled solution leaks, it flagsOf its flags on the 108 labelled tasks, the author labels a leak
Leaklens stated-fix count3535 of 56 (63%); the other 21 are listed
issuelint, issue text99 of 14 (64%)
issuelint, with comments1111 of 17 (65%)

So Leaklens finds about three times as many labelled leaks on the same 108 tasks, with the sentence quoted. The precision is about the same: the share of flags the author labels a leak is close for both tools. The difference is coverage. A fix written in prose, or given in a comment, scores 0 for a code-overlap check, and issuelint’s README says so. Across all 500 Verified tasks, 16 of issuelint’s 19 flags are among the 149 tasks in the Leaklens count (37 of 42 with comments).

The labels cover only the 108 tasks whose patch SWE-Agent with GPT-4 got accepted, one reader per patch, so neither tool’s recall on the other tasks is measured. issuelint also checks things Leaklens does not: whether the issue gives away the tests, whether it names the location, and whether it is too vague to solve.

Audits and tools, side by side

SourceWhat it measuresPer-task results public?What Leaklens does that it does notWhat it does that Leaklens does not
AgentSeal (MIT repo), June 2026 Whether the gold patches or hidden tests of the 731 public SWE-bench Pro tasks are already visible in public code: five exposure signals. It states it does not prove that a model memorised anything. Partly: its report lists 234 task IDs with the repositories that hold copies. Reads the issue text and comments of every Verified and Lite task and quotes any sentence that states a fix. AgentSeal’s signals do not include issue or comment text, and we found no Verified report from it. We ran its signals that need no key on all 500 Verified tasks: its Stack v2 flag marks the repository for 489 of them, and its token overlap hits 18 tasks, none of them among the 43 labelled leaks. Its copy search needs a GitHub token; we did not run it, so we do not say what it finds on Verified. Looks for copies of the gold patch and tests in other repositories, and covers Pro. Leaklens checks Pro only by rule (gold-patch lines in the issue text).
OpenAI on SWE-bench Verified, February 2026 Test quality on 138 tasks its model did not solve consistently, each read by at least six engineers (59.4% flawed), plus a probe in which models recited task text and gold patches. No: six example task IDs. No list of the 138 and no per-task verdicts. A page for each of the 500 tasks, quoting the sentence wherever one states a fix. Measures test quality and what models can recite. Leaklens does neither.
OpenAI on SWE-bench Pro, July 2026 Test and prompt quality of the 731 public Pro tasks: about 30% broken. A quality audit, not a leakage one. No task IDs. Per-task evidence of a fix stated in the task text. Judges whether tests and prompts are sound. Leaklens does not measure test quality.
SWE-Bench+, 2024 A hand review of the patches SWE-Agent with GPT-4 got accepted: solution leaks and weak tests (37 of the 112 Verified tasks it resolved had a solution leak). Partly: the paper gives counts; a separate labels file covers 108 Verified tasks. Covers all 500 Verified tasks, not only those one agent solved, and quotes the sentence wherever one states a fix. A human reader per patch, and a weak-test review. Its labels are what our precision is measured against.
Epoch AI review of Verified, September 2026 One rubric verdict for the whole benchmark (“Flawed”), relying on published audits. No per-task table. Per-task evidence you can check against the issue. A benchmark-level judgement across many kinds of problem, not only leakage.
issuelint (MIT), November 2026 By rule, how much of the gold patch’s added code appears in the issue’s code blocks, plus test leakage, location hints and vagueness. It runs on your machine; no published per-task output for Verified. Our run is above. Fixes stated in prose or in comments: 35 of the 43 labelled leaks against 9 to 11, each with the sentence quoted on a public page. Runs offline with no model on any benchmark file, and checks test leakage and underspecified issues.
BenchChecker, October 2026 Whether a model saw a task’s repository in training (a black-box probe of the model), and whether the fix was merged and still in the repository at the model’s cutoff. Its release labels 450 Verified tasks by whether the patch is in The Stack v2. No detector output for any closed model. Reads the issue text and comments and quotes any sentence that states a fix. For each of the 500 Verified tasks it shows the upstream fix commit, its date and link, and the repository’s public fork count, so you can set the date against any model’s cutoff. BenchChecker publishes no per-task list for any closed model. Probes the model itself, and checks that the fix was still in the repository at the cutoff, which a commit date cannot show. On the 450 Verified tasks its release labels, at The Stack v2’s September 2023 cutoff, the Leaklens timeline says “public exposure possible” for every task with a date (444); the release labels 68 of the 450 patches as not in The Stack v2; at that cutoff the date separates none of them.
arXiv 2512.10218 (Prathifkumar, Mathews and Nagappan), 2025 Gives two Claude models only the issue text and asks which files to edit. Both name every edited file for 65% and 63.2% of the 500 Verified issues, against 12.2% and 12% of 500 issues from five repositories outside Verified. The authors take this to suggest the models may have seen many Verified tasks. We did not check whether it publishes per-task results. Reads the text itself: quotes the sentence in the issue or comments that states the fix, task by task, and shows the upstream fix commit date. Use both: different question. Probes the model, so it speaks to what a model may have seen, which a task’s text cannot show.
LessLeak-Bench (arXiv 2502.06215), Zhou et al., 2025 Matches 83 software-engineering benchmarks against StarCoder’s pre-training data and reports 10.6% leakage for SWE-bench Verified: near-duplicates in a training corpus. We did not check what the release lists per task. Counts tasks whose own issue text or comments state the fix (149 of 500), which a model needs no training to use. Use both: different question. Compares against an actual pre-training corpus and covers 83 benchmarks; Leaklens shows per-task evidence for Verified and Lite.

What Leaklens does not do

  • It does not test what a model memorised or can recite. A flag records what the task text contains.
  • It does not judge whether a task’s tests are sound.
  • It has not measured leakage on SWE-bench Pro beyond the rule check.
  • A fix commit dated before a model’s cutoff means public exposure was possible. It is shown on each task page and never counted as a flag. A date cannot show whether the fix was still in a training corpus at the cutoff; BenchChecker tests that.
  • Precision: the SWE-Bench+ author labels 35 of the 56 labelled tasks in the stated-fix count a leak (63%); the other 21 are listed.

Look up a task Browse all 500 Verified tasks