Notes › 2026-11-16
Where SWE-bench's hints_text stops, checked on 231 Django tasks
SWE-bench's hints_text field is the issue comments posted before the fix pull request's first commit, and later comments are not in it. That is the rule on the dataset card (SWE-bench Verified). Leaklens reads hints_text as a task's "discussion", so where it stops decides what Leaklens reads. We checked the rule against the source threads for the 231 Django tasks in SWE-bench Verified, whose issues live on Django's public Trac tracker.
What we found
- The rule holds for 229 of 231 tasks. For each task we took the Trac ticket's comments posted before the fix pull request's first commit (UTC). That set reproduces
hints_textexactly for 229 tasks. The ticket title is the first line ofproblem_statementfor all 231. - The 2 exceptions. In django__django-13315 the pull request fixes two tickets (#1891 and #11707), and
hints_textholds only #11707's thread. In django__django-16032 one comment was edited on Trac after the dataset copied it. - Short comments set aside. We set aside 135 comments with fewer than 12 letters or digits, mostly the 3-character "PR" link comment, because a containment test finds them inside almost any text. Counted, they made 51 tasks look mismatched, each because of such a comment.
- No timezone shift. If Trac's displayed US Central clock is read as UTC, the rule matches only 117 of 231 tasks, so the cutoff is not shifted by a timezone.
What the cutoff leaves out: django__django-11211
The fix pull request's first commit is 2019-04-10 05:13:16 UTC. Three Trac comments that give the fix as a diff were posted minutes later (05:16, 05:26 and 05:37 UTC), and the pull request opened on 2019-04-13. So hints_text holds only the first comment, by SWE-bench's rule rather than a dropped comment. A model that is shown hints_text never sees those comments. A model trained on the public web could have.
So the task pages now show this as exposure, never as a flag, with the tier unchanged. Where Trac text posted after the dataset's cutoff and before the fix commit repeats the gold patch's added lines (Leaklens's Tier C rule, unchanged), the task's timeline gains a line with the comment's date, a link to it and how many added lines it repeats. The "public from" date then takes the earlier of that comment and the fix commit. Of the 231 Django tasks, 144 have Trac text posted before the fix commit that is not in hints_text, and in 3 of them that text alone meets the rule:
| Task | Trac comment | Added lines repeated | Fix commit |
|---|---|---|---|
| django__django-11211 | #30343 comment 3, 2019-04-10 | 3 of 3 | 2019-04-14 |
| django__django-11999 | #30931 comment 15, 2019-10-31 | 2 of 3 | 2019-11-04 |
| django__django-13658 | #32177 comment 3, 2020-11-09 | 3 of 5 | 2020-11-10 |
Three tasks have no fix-commit date and were not checked. None of this changes a tier or the Verified headline of 149 of 500.
The other benchmarks
Each benchmark Leaklens lists ships its own field, or none. The methodology table names it for each one. SWE-bench Pro and Multi-SWE-bench ship no comment field, so only the issue text is read. For SWE-rebench, SWE-Gym, SWE-PolyBench and OmniCode, hints_text is read as given, and where it stops has not been checked yet.
Repeat it
The counts above come from a cache of the 231 Trac ticket pages and fix pull-request patches, fetched anonymously at one request a second on 16 Nov 2026. That cache and the scripts that read it are in Leaklens's working repository, not yet in the public engine repository. Until they are, you can repeat any row by hand: open the task's Trac ticket, note the fix pull request's first commit time on GitHub, and compare the comments posted before it with the task's hints_text.
Cite this note
Leaklens, “Where SWE-bench's hints_text stops, checked on 231 Django tasks”, https://leaklens.grain64.com/notes/swe-bench-hints-text-cutoff, published 2026-11-16. Table: https://leaklens.grain64.com/benchmarks/swe-bench-verified, methodology m1.1.
BibTeX
@misc{leaklens_swe_bench_hints_text_cutoff,
title = {Where SWE-bench's hints_text stops, checked on 231 Django tasks},
author = {{Leaklens}},
year = {2026},
doi = {10.5281/zenodo.23990163},
url = {https://leaklens.grain64.com/notes/swe-bench-hints-text-cutoff},
note = {Table: https://leaklens.grain64.com/benchmarks/swe-bench-verified, methodology m1.1}
}