SWE-Bench Pro (public sample): every task, with its verdict
SWE-Bench Pro, published by Scale AI. Paper authors: Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu and Brad Kenstler. SWE-Bench Pro (arXiv) · This page covers a random 100-task sample of the 731 public tasks (v1 config of the ScaleAI/SWE-bench_Pro dataset). The sample is the 100 instance IDs with the lowest SHA-256 of "own1:swe-bench-pro:<instance_id>", so anyone can redraw it. · Report a correction · flags show what the public text and git history contain: public exposure was possible, nothing more.
| Task | Verdict | Issue title |
|---|---|---|
| instance_ansible__ansible-1c06c46cc14324df35ac4f39a45fb3ccd602195d-v0f01c69f1e2528b935359cfe578530722bca2c59 | no overlap found (rule check) | Fix YAML filter trust propagation and vault handling |
| instance_ansible__ansible-489156378c8e97374a75a544c7c9c2c0dd8146d1-v390e508d27db7a51eece36bb6d9698b63a5b638a | no overlap found (rule check) | Meraki modules fail immediately on HTTP 429/500/502 responses from the Meraki API |
| instance_ansible__ansible-5e88cd9972f10b66dd97e1ee684c910c6a2dd25e-v906c969b551b346ef54a2c0b41e04f632b7b73c2 | no overlap found (rule check) | Missing Ansible module for user management on Pluribus Networks devices. |
| instance_ansible__ansible-6cc97447aac5816745278f3735af128afb255c81-v0f01c69f1e2528b935359cfe578530722bca2c59 | no overlap found (rule check) | ansible-core: Inconsistent behavior with unset values, deprecations, `None` overrides in templar, legacy YAML constructors, lookup messages, and CLI errors |
| instance_ansible__ansible-8127abbc298cabf04aaa89a478fc5e5e3432a6fc-v30a923fb5c164d6cd18280c02422f75e611e8fb2 | no overlap found (rule check) | Isolate worker processes by detaching inherited standard I/O to prevent unintended terminal interaction. |
| instance_ansible__ansible-83fb24b923064d3576d473747ebbe62e4535c9e3-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5 | no overlap found (rule check) | Lack of support for multiple destination ports in the iptables module |
| instance_ansible__ansible-a20a52701402a12f91396549df04ac55809f68e9-v1055803c3a812189a1133297f7f5468579283f86 | no overlap found (rule check) | ansible-galaxy allows path traversal when installing collections from malicious tar files |
| instance_ansible__ansible-b5e0293645570f3f404ad1dbbe5f006956ada0df-v0f01c69f1e2528b935359cfe578530722bca2c59 | no overlap found (rule check) | PowerShell CLIXML output displays escaped sequences instead of actual characters |
| instance_ansible__ansible-c616e54a6e23fa5616a1d56d243f69576164ef9b-v1055803c3a812189a1133297f7f5468579283f86 | no overlap found (rule check) | `module_common` fails to resolve `module_utils` from collections (redirects, package `__init__` relative imports) and shows confusing errors |
| instance_ansible__ansible-cb94c0cc550df9e98f1247bc71d8c2b861c75049-v1055803c3a812189a1133297f7f5468579283f86 | no overlap found (rule check) | Missing `timeout` in ad-hoc and console CLIs; `task_include` ignores `timeout`; console lacks extra-vars option |
| instance_ansible__ansible-d30fc6c0b359f631130b0e979d9a78a7b3747d48-v1055803c3a812189a1133297f7f5468579283f86 | no overlap found (rule check) | ansible-galaxy does not preserve internal symlinks in collections and lacks safe extraction for symlink members |
| instance_ansible__ansible-d58e69c82d7edd0583dd8e78d76b075c33c3151e-v173091e2e36d38c978002990795f66cfc0af30ad | no overlap found (rule check) | `uri` and `get_url` modules fail to handle gzip-encoded HTTP responses |
| instance_ansible__ansible-d9f1866249756efc264b00ff7497e92c11a9885f-v0f01c69f1e2528b935359cfe578530722bca2c59 | no overlap found (rule check) | Deprecate the usage of `safe_eval` in Ansible module_utils |
| instance_ansible__ansible-de01db08d00c8d2438e1ba5989c313ba16a145b0-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5 | no overlap found (rule check) | pip module fails when `executable` and `virtualenv` are unset and no `pip` binary is found |
| instance_ansible__ansible-e40889e7112ae00a21a2c74312b330e67a766cc0-v1055803c3a812189a1133297f7f5468579283f86 | no overlap found (rule check) | Support specifying collections in git repositories in requirements.yml |
| instance_ansible__ansible-f8ef34672b961a95ec7282643679492862c688ec-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5 | no overlap found (rule check) | Preserve YAML object context (`.obj`) for errors raised when loading tasks / decrypting single-value Ansible Vault scalars. |