SWE-Bench Pro (public sample): every task, with its verdict
SWE-Bench Pro, published by Scale AI. Paper authors: Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu and Brad Kenstler. SWE-Bench Pro (arXiv) · This page covers a random 100-task sample of the 731 public tasks (v1 config of the ScaleAI/SWE-bench_Pro dataset). The sample is the 100 instance IDs with the lowest SHA-256 of "own1:swe-bench-pro:<instance_id>", so anyone can redraw it. · Report a correction · flags show what the public text and git history contain: public exposure was possible, nothing more.
| Task | Verdict | Issue title |
|---|---|---|
| instance_NodeBB__NodeBB-05f2236193f407cf8e2072757fbd6bb170bc13f0-vf2cf3cbd463b7ad942381f1c6d077626485a1e9e | no overlap found (rule check) | Lack of support for retrieving topics in ascending order by last post date |
| instance_NodeBB__NodeBB-82562bec444940608052f3e4149e0c61ec80bf3f-vd59a5728dfc977f44533186ace531248c2917516 | no overlap found (rule check) | Upvoter list can be fetched without required read privileges |
| instance_NodeBB__NodeBB-84dfda59e6a0e8a77240f939a7cb8757e6eaf945-v2c59007b1005cd5cd14cbb523ca5229db1fd2dd8 | no overlap found (rule check) | Automatic deletion of uploaded files when purging a post |
| instance_NodeBB__NodeBB-97c8569a798075c50e93e585ac741ab55cb7c28b-vf2cf3cbd463b7ad942381f1c6d077626485a1e9e | no overlap found (rule check) | User API Returns Private Fields Without Proper Filtering |
| instance_NodeBB__NodeBB-be43cd25974681c9743d424238b7536c357dc8d3-vf2cf3cbd463b7ad942381f1c6d077626485a1e9e | no overlap found (rule check) | Feature: Reverse links to topics |
| instance_NodeBB__NodeBB-f083cd559d69c16481376868c8da65172729c0ca-vnan | no overlap found (rule check) | Add support for retrieving sorted-set members *with scores` in the database layer |
| instance_ansible__ansible-1c06c46cc14324df35ac4f39a45fb3ccd602195d-v0f01c69f1e2528b935359cfe578530722bca2c59 | no overlap found (rule check) | Fix YAML filter trust propagation and vault handling |
| instance_ansible__ansible-489156378c8e97374a75a544c7c9c2c0dd8146d1-v390e508d27db7a51eece36bb6d9698b63a5b638a | no overlap found (rule check) | Meraki modules fail immediately on HTTP 429/500/502 responses from the Meraki API |
| instance_ansible__ansible-5e88cd9972f10b66dd97e1ee684c910c6a2dd25e-v906c969b551b346ef54a2c0b41e04f632b7b73c2 | no overlap found (rule check) | Missing Ansible module for user management on Pluribus Networks devices. |
| instance_ansible__ansible-6cc97447aac5816745278f3735af128afb255c81-v0f01c69f1e2528b935359cfe578530722bca2c59 | no overlap found (rule check) | ansible-core: Inconsistent behavior with unset values, deprecations, `None` overrides in templar, legacy YAML constructors, lookup messages, and CLI errors |
| instance_ansible__ansible-8127abbc298cabf04aaa89a478fc5e5e3432a6fc-v30a923fb5c164d6cd18280c02422f75e611e8fb2 | no overlap found (rule check) | Isolate worker processes by detaching inherited standard I/O to prevent unintended terminal interaction. |
| instance_ansible__ansible-83fb24b923064d3576d473747ebbe62e4535c9e3-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5 | no overlap found (rule check) | Lack of support for multiple destination ports in the iptables module |
| instance_ansible__ansible-a20a52701402a12f91396549df04ac55809f68e9-v1055803c3a812189a1133297f7f5468579283f86 | no overlap found (rule check) | ansible-galaxy allows path traversal when installing collections from malicious tar files |
| instance_ansible__ansible-b5e0293645570f3f404ad1dbbe5f006956ada0df-v0f01c69f1e2528b935359cfe578530722bca2c59 | no overlap found (rule check) | PowerShell CLIXML output displays escaped sequences instead of actual characters |
| instance_ansible__ansible-c616e54a6e23fa5616a1d56d243f69576164ef9b-v1055803c3a812189a1133297f7f5468579283f86 | no overlap found (rule check) | `module_common` fails to resolve `module_utils` from collections (redirects, package `__init__` relative imports) and shows confusing errors |
| instance_ansible__ansible-cb94c0cc550df9e98f1247bc71d8c2b861c75049-v1055803c3a812189a1133297f7f5468579283f86 | no overlap found (rule check) | Missing `timeout` in ad-hoc and console CLIs; `task_include` ignores `timeout`; console lacks extra-vars option |
| instance_ansible__ansible-d30fc6c0b359f631130b0e979d9a78a7b3747d48-v1055803c3a812189a1133297f7f5468579283f86 | no overlap found (rule check) | ansible-galaxy does not preserve internal symlinks in collections and lacks safe extraction for symlink members |
| instance_ansible__ansible-d58e69c82d7edd0583dd8e78d76b075c33c3151e-v173091e2e36d38c978002990795f66cfc0af30ad | no overlap found (rule check) | `uri` and `get_url` modules fail to handle gzip-encoded HTTP responses |
| instance_ansible__ansible-d9f1866249756efc264b00ff7497e92c11a9885f-v0f01c69f1e2528b935359cfe578530722bca2c59 | no overlap found (rule check) | Deprecate the usage of `safe_eval` in Ansible module_utils |
| instance_ansible__ansible-de01db08d00c8d2438e1ba5989c313ba16a145b0-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5 | no overlap found (rule check) | pip module fails when `executable` and `virtualenv` are unset and no `pip` binary is found |
| instance_ansible__ansible-e40889e7112ae00a21a2c74312b330e67a766cc0-v1055803c3a812189a1133297f7f5468579283f86 | no overlap found (rule check) | Support specifying collections in git repositories in requirements.yml |
| instance_ansible__ansible-f8ef34672b961a95ec7282643679492862c688ec-vba6da65a0f3baefda7a058ebbd0a8dcafb8512f5 | no overlap found (rule check) | Preserve YAML object context (`.obj`) for errors raised when loading tasks / decrypting single-value Ansible Vault scalars. |
| instance_element-hq__element-web-1077729a19c0ce902e713cf6fab42c91fb7907f1-vnan | no overlap found (rule check) | New Room List: Prevent potential scroll jump/flicker when switching spaces |
| instance_element-hq__element-web-4c6b0d35add7ae8d58f71ea1711587e31081444b-vnan | no overlap found (rule check) | PosthogAnalytics fails to reliably handle initialization, anonymity, and event tracking under different configuration and privacy scenarios |
| instance_element-hq__element-web-53b42e321777a598aaf2bb3eab22d710569f83a8-vnan | no overlap found (rule check) | A way to prevent displaying the room options menu |
| instance_element-hq__element-web-582a1b093fc0b77538052f45cbb9c7295f991b51-vnan | no overlap found (rule check) | Limit decryption failure tracking to visible events and reduce reporting delay ## Description The decryption failure tracking system currently observes all events with decryption errors, regardless of their visibility in the UI. This results in unnecessary tracking of events that users may never see |
| instance_element-hq__element-web-6205c70462e0ce2e1e77afb3a70b55d0fdfe1b31-vnan | no overlap found (rule check) | Voice Broadcast Liveness Does Not Match Broadcast Info State |
| instance_element-hq__element-web-72a8f8f03b1a01bb70ef8a5bb61759416991b32c-vnan | no overlap found (rule check) | Provide a way to read current window width from UI state |
| instance_element-hq__element-web-772df3021201d9c73835a626df8dcb6334ad9a3e-vnan | no overlap found (rule check) | Lack of Multi-Selection Support for Device Sign-Out. |
| instance_element-hq__element-web-880428ab94c6ea98d3d18dcaeb17e8767adcb461-vnan | no overlap found (rule check) | Improve toast notifications and actions for new device logins. |
| instance_element-hq__element-web-9a31cd0fa849da810b4fac6c6c015145e850b282-vnan | no overlap found (rule check) | Issue Title: Allow setting room join rule to "knock" ## What would you like to do? Add a feature-flagged “Ask to join” (Knock) join rule to Room Settings: show it only when feature_ask_to_join is enabled, and if the current room version doesn’t support Knock, show the standard upgrade prompt (with p |
| instance_element-hq__element-web-a692fe21811f88d92e8f7047fc615e4f1f986b0f-vnan | no overlap found (rule check) | Problem Statement |
| instance_element-hq__element-web-aeabf3b18896ac1eb7ae9757e66ce886120f8309-vnan | no overlap found (rule check) | Lack of message type context in the Thread list (roots and replies), plus duplicated preview logic. |
| instance_element-hq__element-web-dae13ac8522fc6d41e64d1ac6e3174486fdcce0c-vnan | no overlap found (rule check) | Unread indicators diverge between room and thread timelines |
| instance_element-hq__element-web-e15ef9f3de36df7f318c083e485f44e1de8aad17 | no overlap found (rule check) | Missing independent device-level notification toggle |
| instance_element-hq__element-web-ecfd1736e5dd9808e87911fc264e6c816653e1a9-vnan | no overlap found (rule check) | Combine search results when the query is present in multiple successive messages |
| instance_element-hq__element-web-ee13e23b156fbad9369d6a656c827b6444343d4f-vnan | no overlap found (rule check) | `RoomHeaderButtons` can crash when thread notifications are unsupported or when the `room` prop is missing. |
| instance_element-hq__element-web-f14374a51c153f64f313243f2df6ea4971db4e15 | no overlap found (rule check) | Improve Message Composer Component Visibility |
| instance_element-hq__element-web-f3534b42df3dcfe36dc48bddbf14034085af6d30-vnan | no overlap found (rule check) | Membership event combining display name and profile picture changes lacks a single descriptive message |
| instance_element-hq__element-web-f63160f38459fb552d00fcc60d4064977a9095a6-vnan | no overlap found (rule check) | Inconsistent and unclear display of key verification requests in timeline |
| instance_flipt-io__flipt-2ca5dfb3513e4e786d2b037075617cccc286d5c3 | no overlap found (rule check) | Support multiple metrics exporters (Prometheus, OpenTelemetry) |
| instance_flipt-io__flipt-2eac0df47b5ecc8bb05002d80383ceb08ab3620a | no overlap found (rule check) | Telemetry payload doesn't reflect analytics state or backend and carries an outdated payload version identifier |
| instance_flipt-io__flipt-756f00f79ba8abf9fe53f3c6c818123b42eb7355 | no overlap found (rule check) | Configuration refactoring to separate warnings from Config and deprecate ui.enabled |
| instance_flipt-io__flipt-86906cbfc3a5d3629a583f98e6301142f5f14bdb-v6bea0cc3a6fc532d7da914314f2944fc1cd04dee | no overlap found (rule check) | Snapshot cache does not allow controlled deletion of references |
| instance_flipt-io__flipt-aebaecd026f752b187f11328b0d464761b15d2ab | no overlap found (rule check) | Git backend fails to poll when a reference no longer exists |
| instance_flipt-io__flipt-b2cd6a6dd73ca91b519015fd5924fde8d17f3f06 | no overlap found (rule check) | Telemetry warns about non-writable state directory in read-only environments |
| instance_flipt-io__flipt-c12967bc73fdf02054cf3ef8498c05e25f0a18c0 | no overlap found (rule check) | Context cancellation and deadline exceeded errors are incorrectly classified as internal in gRPC responses. |
| instance_flipt-io__flipt-c6a7b1fd933e763b1675281b30077e161fa115a1 | no overlap found (rule check) | Add namespace and version metadata to export files; validate on import |
| instance_flipt-io__flipt-f36bd61fb1cee4669de1f00e59da462bfeae8765 | no overlap found (rule check) | `flipt validate` produces imprecise and repetitive error messages when validating YAML files. |
| instance_future-architect__vuls-01441351c3407abfc21c48a38e28828e1b504e0c | no overlap found (rule check) | SNMP2CPE fails to emit correct CPEs for Fortinet – FortiSwitch-108E case |
| instance_future-architect__vuls-030b2e03525d68d74cb749959aac2d7f3fc0effa | no overlap found (rule check) | Outdated security‑update mapping for certain Windows releases in the Vuls scanner. |
| instance_future-architect__vuls-5af1a227339e46c7abf3f2815e4c636a0c01098e | no overlap found (rule check) | Issue Title: Incorrect detection of running kernel package versions when multiple variants are installed |
| instance_future-architect__vuls-dc496468b9e9fb73371f9606cdcdb0f8e12e70ca | no overlap found (rule check) | Reorganize syslog configuration into a dedicated configuration component. |
| instance_future-architect__vuls-f6cc8c263dc00329786fa516049c60d4779c4a07 | no overlap found (rule check) | Package name parsing produces incorrect namespace, name, or subpath in PURLs |
| instance_gravitational__teleport-0415e422f12454db0c22316cf3eaa5088d6b6322 | no overlap found (rule check) | Multi-Device U2F Authentication Restricted to Single Token Selection |
| instance_gravitational__teleport-46a13210519461c7cec8d643bfbe750265775b41 | no overlap found (rule check) | `tctl auth sign --format=kubernetes` uses incorrect port from proxy public address |
| instance_gravitational__teleport-4f771403dc4177dc26ee0370f7332f3fe54bee0f-vee9b09fb20c43af7e520f57e9239bbcf46b7113d | no overlap found (rule check) | Foundational buffering and deadline primitives for resilient connections |
| instance_gravitational__teleport-ac2fb2f9b4fd1896b554d3011df23d3d71295779 | no overlap found (rule check) | Auth service crashing |
| instance_gravitational__teleport-b1bcd8b90c474a35bb11cc3ef4cc8941e1f8eab2-vee9b09fb20c43af7e520f57e9239bbcf46b7113d | no overlap found (rule check) | Incorrect counting of authenticated HTTP connections in ingress reporter metrics |
| instance_gravitational__teleport-c335534e02de143508ebebc7341021d7f8656e8f | no overlap found (rule check) | Issues with certificate validation in tsh proxy ssh |
| instance_gravitational__teleport-dd3977957a67bedaf604ad6ca255ba8c7b6704e9 | no overlap found (rule check) | Support additional principals for Teleport services. |
| instance_gravitational__teleport-f432a71a13e698b6e1c4672a2e9e9c1f32d35c12 | no overlap found (rule check) | Add KeyStore interface and rawKeyStore implementation to manage cryptographic keys |
| instance_internetarchive__openlibrary-0dc5b20fa186f9714f8a838178597e69f549d026-v2d9a6c849c60ed19fd0858ce9e40b7cc8e097e59 | no overlap found (rule check) | Import alternate-script author names |
| instance_internetarchive__openlibrary-1351c59fd43689753de1fca32c78d539a116ffc1-v29f82c9cf21d57b242f8d8b0e541525d259e2d63 | no overlap found (rule check) | Inconsistency in author identifier generation when comparing editions. |
| instance_internetarchive__openlibrary-7bf3238533070f2d24bafbb26eedf675d51941f6-v08d8e8889ec945ab821fb156c04c7d2e2810debb | no overlap found (rule check) | Add Reading-Log Counts to Solr Work Documents |
| instance_internetarchive__openlibrary-7c8dc180281491ccaa1b4b43518506323750d1e4-v298a7a812ceed28c4c18355a091f1b268fe56d86 | no overlap found (rule check) | Function `read_subjects()` in `get_subjects.py` exceeds acceptable complexity thresholds and includes unused logic |
| instance_internetarchive__openlibrary-8a5a63af6e0be406aa6c8c9b6d5f28b2f1b6af5a-v0f5aece3601a5b4419f7ccec1dbda2071be28ee4 | no overlap found (rule check) | Host-scoped scheduling for background jobs |
| instance_internetarchive__openlibrary-8a9d9d323dfcf2a5b4f38d70b1108b030b20ebf3-v13642507b4fc1f8d234172bf8129942da2c2ca26 | no overlap found (rule check) | Support importing staged ISBNdb data dumps via CLI |
| instance_internetarchive__openlibrary-ba3abfb6af6e722185d3715929ab0f3e5a134eed-v76304ecdb3a5954fcf13feb710e8c40fcf24b73c | no overlap found (rule check) | Allow Import API to Bypass Validation Checks via `override-validation` Flag ## Description **Label:** Feature Request **Problem / Opportunity** The current book import process fails when validation rules are triggered, such as for books published too far in the past or future, those without ISBNs wh |
| instance_internetarchive__openlibrary-d40ec88713dc95ea791b252f92d2f7b75e107440-v13642507b4fc1f8d234172bf8129942da2c2ca26 | no overlap found (rule check) | Add preview option to import endpoints and clarify import validation behavior |
| instance_internetarchive__openlibrary-e390c1212055dd84a262a798e53487e771d3fb64-v8717e18970bcdc4e0d2cea3b1527752b21e74866 | no overlap found (rule check) | Incorrect availability prioritization and metadata for multi-edition IA works in Solr documents |
| instance_navidrome__navidrome-3977ef6e0f287f598b6e4009876239d6f13b686d | no overlap found (rule check) | Hasher lacks deterministic seeding needed for stable “random” ordering |
| instance_navidrome__navidrome-55bff343cdaad1f04496f724eda4b55d422d7f17 | no overlap found (rule check) | Simplify SQLite3 access by reverting read/write separation |
| instance_navidrome__navidrome-56303cde23a4122d2447cbb266f942601a78d7e4 | no overlap found (rule check) | Scanner does not support R128 gain tags for track and album |
| instance_navidrome__navidrome-69e0a266f48bae24a11312e9efbe495a337e4c84 | no overlap found (rule check) | Issue Title: Remove size from public image ID JWT. |
| instance_navidrome__navidrome-6bd4c0f6bfa653e9b8b27cfdc2955762d371d6e9 | no overlap found (rule check) | Possible to remove authentication? |
| instance_navidrome__navidrome-89b12b34bea5687c70e4de2109fd1e7330bb2ba2 | no overlap found (rule check) | Last.fm API not getting correct Artist info |
| instance_protonmail__webclients-01b519cd49e6a24d9a05d2eb97f54e420740072e | no overlap found (rule check) | Support for HEIC/JXL thumbnail and preview generation in MacOS Safari 17+ |
| instance_protonmail__webclients-2c3559cad02d1090985dba7e8eb5a129144d9811 | no overlap found (rule check) | Assistant upsell inconsistency from outdated addon resolver. |
| instance_protonmail__webclients-51742625834d3bd0d10fe0c7e76b8739a59c6b9f | no overlap found (rule check) | Implement proper Punycode encoding for URLs to prevent IDN phishing attacks |
| instance_protonmail__webclients-944adbfe06644be0789f59b78395bdd8567d8547 | no overlap found (rule check) | API error metrics |
| instance_protonmail__webclients-d3e513044d299d04e509bf8c0f4e73d812030246 | no overlap found (rule check) | Standardizing mail metrics helper functions. |
| instance_qutebrowser__qutebrowser-0fc6d1109d041c69a68a896db87cf1b8c194cef7-v2ef375ac784985212b1805e1d0431dc8f1b3c171 | no overlap found (rule check) | Add filesystem path completion support for `:open` command |
| instance_qutebrowser__qutebrowser-1943fa072ec3df5a87e18a23b0916f134c131016-vafb3e8e01b31319c66c4e666b8a3b1d8ba55db24 | no overlap found (rule check) | Handle tab pinned status in AbstractTab |
| instance_qutebrowser__qutebrowser-2dd8966fdcf11972062c540b7a787e4d0de8d372-v363c8a7e5ccdf6968fc7ab84a2053ac78036691d | no overlap found (rule check) | Need utility function to convert QColor objects to QSS-compatible color strings |
| instance_qutebrowser__qutebrowser-2e961080a85d660148937ee8f0f6b3445a8f2c01-v363c8a7e5ccdf6968fc7ab84a2053ac78036691d | no overlap found (rule check) | Inconsistent handling of `qt.workarounds.disable_accelerated_2d_canvas` option across versions. |
| instance_qutebrowser__qutebrowser-3fd8e12949b8feda401930574facf09dd4180bba | no overlap found (rule check) | Standardize command naming with consistent cmd- prefix |
| instance_qutebrowser__qutebrowser-6dd402c0d0f7665d32a74c43c5b4cf5dc8aff28d-v5fc38aaf22415ab0b70567368332beee7955b367 | no overlap found (rule check) | Application Crashes When Adblock Cache File is Corrupted |
| instance_qutebrowser__qutebrowser-7f9713b20f623fc40473b7167a082d6db0f0fd40-va0fd88aac89cde702ec1ba84877234da33adce8a | no overlap found (rule check) | JPG files don't show up in file picker when filetypes are restricted to images |
| instance_qutebrowser__qutebrowser-e34dfc68647d087ca3175d9ad3f023c30d8c9746-v363c8a7e5ccdf6968fc7ab84a2053ac78036691d | no overlap found (rule check) | URL parsing and search term handling edge cases cause incorrect behavior in `urlutils.py` |
| instance_qutebrowser__qutebrowser-e5340c449f23608803c286da0563b62f58ba25b0-v059c6fdc75567943479b23ebca7c07b5e9a7f34c | no overlap found (rule check) | WebKit Certificate Error Wrapper Has Inconsistent Constructor and HTML Rendering |
| instance_qutebrowser__qutebrowser-e57b6e0eeeb656eb2c84d6547d5a0a7333ecee85-v2ef375ac784985212b1805e1d0431dc8f1b3c171 | no overlap found (rule check) | BlocklistDownloads Uses Outdated Callback Pattern Instead of Qt Signals |
| instance_qutebrowser__qutebrowser-e70f5b03187bdd40e8bf70f5f3ead840f52d1f42-v02ad04386d5238fe2d1a1be450df257370de4b6a | no overlap found (rule check) | Required message's improvements for process |
| instance_tutao__tutanota-09c2776c0fce3db5c6e18da92b5a45dce9f013aa-vbc0d9ba8f0071fbe982809910959a6ff8884dbbf | no overlap found (rule check) | Lack of progress tracking during calendar imports |
| instance_tutao__tutanota-8513a9e8114a8b42e64f4348335e0f23efa054c4-vee878bb72091875e912c52fc32bc60ec3760227b | no overlap found (rule check) | Device Configuration Overwrite on Load |
| instance_tutao__tutanota-d1aa0ecec288bfc800cfb9133b087c4f81ad8b38-vbc0d9ba8f0071fbe982809910959a6ff8884dbbf | no overlap found (rule check) | Draft Mail Validation Ignores Subfolder Hierarchy in Drafts Folder |
| instance_tutao__tutanota-da4edb7375c10f47f4ed3860a591c5e6557f7b5c-vbc0d9ba8f0071fbe982809910959a6ff8884dbbf | no overlap found (rule check) | Blob read token request requires unnecessary `archiveDataType` in owned archive scenarios |
| instance_tutao__tutanota-f373ac3808deefce8183dad8d16729839cc330c1-v2939aa9f4356f0dc9f523ee5ce19d09e08ab979b | no overlap found (rule check) | Owner-encrypted session key is not propagated through loaders and cache, causing Mail details to fail decryption |
| instance_tutao__tutanota-fb32e5f9d9fc152a00144d56dd0af01760a2d4dc-vc4e41fd0029957297843cb9dec4a25c7c756f029 | no overlap found (rule check) | vCard export outputs vanity handles and escapes “:” in URLs, producing invalid links and inconsistency with the web client |
| instance_tutao__tutanota-fbdb72a2bd39b05131ff905780d9d4a2a074de26-vbc0d9ba8f0071fbe982809910959a6ff8884dbbf | no overlap found (rule check) | Referral links and settings are visible to business customers who are not eligible to use them. |