Leaklens

Notes › 2026-11-16

Where SWE-bench's hints_text stops, checked on 231 Django tasks

SWE-bench's hints_text field is the issue comments posted before the fix pull request's first commit, and later comments are not in it. That is the rule on the dataset card (SWE-bench Verified). Leaklens reads hints_text as a task's "discussion", so where it stops decides what Leaklens reads. We checked the rule against the source threads for the 231 Django tasks in SWE-bench Verified, whose issues live on Django's public Trac tracker.

What we found

What the cutoff leaves out: django__django-11211

The fix pull request's first commit is 2019-04-10 05:13:16 UTC. Three Trac comments that give the fix as a diff were posted minutes later (05:16, 05:26 and 05:37 UTC), and the pull request opened on 2019-04-13. So hints_text holds only the first comment, by SWE-bench's rule rather than a dropped comment. A model that is shown hints_text never sees those comments. A model trained on the public web could have.

So the task pages now show this as exposure, never as a flag, with the tier unchanged. Where Trac text posted after the dataset's cutoff and before the fix commit repeats the gold patch's added lines (Leaklens's Tier C rule, unchanged), the task's timeline gains a line with the comment's date, a link to it and how many added lines it repeats. The "public from" date then takes the earlier of that comment and the fix commit. Of the 231 Django tasks, 144 have Trac text posted before the fix commit that is not in hints_text, and in 3 of them that text alone meets the rule:

| Task | Trac comment | Added lines repeated | Fix commit |
|---|---|---|---|
| django__django-11211 | #30343 comment 3, 2019-04-10 | 3 of 3 | 2019-04-14 |
| django__django-11999 | #30931 comment 15, 2019-10-31 | 2 of 3 | 2019-11-04 |
| django__django-13658 | #32177 comment 3, 2020-11-09 | 3 of 5 | 2020-11-10 |

Three tasks have no fix-commit date and were not checked. None of this changes a tier or the Verified headline of 149 of 500.

The other benchmarks

Each benchmark Leaklens lists ships its own field, or none. The methodology table names it for each one. SWE-bench Pro and Multi-SWE-bench ship no comment field, so only the issue text is read. For SWE-rebench, SWE-Gym, SWE-PolyBench and OmniCode, hints_text is read as given, and where it stops has not been checked yet.

Repeat it

The counts above come from a cache of the 231 Trac ticket pages and fix pull-request patches, fetched anonymously at one request a second on 16 Nov 2026. That cache and the scripts that read it are in Leaklens's working repository, not yet in the public engine repository. Until they are, you can repeat any row by hand: open the task's Trac ticket, note the fix pull request's first commit time on GitHub, and compare the comments posted before it with the task's hints_text.

Cite this note

Leaklens, “Where SWE-bench's hints_text stops, checked on 231 Django tasks”, https://leaklens.grain64.com/notes/swe-bench-hints-text-cutoff, published 2026-11-16. Table: https://leaklens.grain64.com/benchmarks/swe-bench-verified, methodology m1.1.

BibTeX
@misc{leaklens_swe_bench_hints_text_cutoff,
  title = {Where SWE-bench's hints_text stops, checked on 231 Django tasks},
  author = {{Leaklens}},
  year = {2026},
  doi = {10.5281/zenodo.23990163},
  url = {https://leaklens.grain64.com/notes/swe-bench-hints-text-cutoff},
  note = {Table: https://leaklens.grain64.com/benchmarks/swe-bench-verified, methodology m1.1}
}