SWE-Bench Pro (public sample): every task, with its verdict
SWE-Bench Pro, published by Scale AI. Paper authors: Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu and Brad Kenstler. SWE-Bench Pro (arXiv) · This page covers a random 100-task sample of the 731 public tasks (v1 config of the ScaleAI/SWE-bench_Pro dataset). The sample is the 100 instance IDs with the lowest SHA-256 of "own1:swe-bench-pro:<instance_id>", so anyone can redraw it. · Report a correction · flags show what the public text and git history contain: public exposure was possible, nothing more.
| Task | Verdict | Issue title |
|---|---|---|
| instance_element-hq__element-web-1077729a19c0ce902e713cf6fab42c91fb7907f1-vnan | no overlap found (rule check) | New Room List: Prevent potential scroll jump/flicker when switching spaces |
| instance_element-hq__element-web-4c6b0d35add7ae8d58f71ea1711587e31081444b-vnan | no overlap found (rule check) | PosthogAnalytics fails to reliably handle initialization, anonymity, and event tracking under different configuration and privacy scenarios |
| instance_element-hq__element-web-53b42e321777a598aaf2bb3eab22d710569f83a8-vnan | no overlap found (rule check) | A way to prevent displaying the room options menu |
| instance_element-hq__element-web-582a1b093fc0b77538052f45cbb9c7295f991b51-vnan | no overlap found (rule check) | Limit decryption failure tracking to visible events and reduce reporting delay ## Description The decryption failure tracking system currently observes all events with decryption errors, regardless of their visibility in the UI. This results in unnecessary tracking of events that users may never see |
| instance_element-hq__element-web-6205c70462e0ce2e1e77afb3a70b55d0fdfe1b31-vnan | no overlap found (rule check) | Voice Broadcast Liveness Does Not Match Broadcast Info State |
| instance_element-hq__element-web-72a8f8f03b1a01bb70ef8a5bb61759416991b32c-vnan | no overlap found (rule check) | Provide a way to read current window width from UI state |
| instance_element-hq__element-web-772df3021201d9c73835a626df8dcb6334ad9a3e-vnan | no overlap found (rule check) | Lack of Multi-Selection Support for Device Sign-Out. |
| instance_element-hq__element-web-880428ab94c6ea98d3d18dcaeb17e8767adcb461-vnan | no overlap found (rule check) | Improve toast notifications and actions for new device logins. |
| instance_element-hq__element-web-9a31cd0fa849da810b4fac6c6c015145e850b282-vnan | no overlap found (rule check) | Issue Title: Allow setting room join rule to "knock" ## What would you like to do? Add a feature-flagged “Ask to join” (Knock) join rule to Room Settings: show it only when feature_ask_to_join is enabled, and if the current room version doesn’t support Knock, show the standard upgrade prompt (with p |
| instance_element-hq__element-web-a692fe21811f88d92e8f7047fc615e4f1f986b0f-vnan | no overlap found (rule check) | Problem Statement |
| instance_element-hq__element-web-aeabf3b18896ac1eb7ae9757e66ce886120f8309-vnan | no overlap found (rule check) | Lack of message type context in the Thread list (roots and replies), plus duplicated preview logic. |
| instance_element-hq__element-web-dae13ac8522fc6d41e64d1ac6e3174486fdcce0c-vnan | no overlap found (rule check) | Unread indicators diverge between room and thread timelines |
| instance_element-hq__element-web-e15ef9f3de36df7f318c083e485f44e1de8aad17 | no overlap found (rule check) | Missing independent device-level notification toggle |
| instance_element-hq__element-web-ecfd1736e5dd9808e87911fc264e6c816653e1a9-vnan | no overlap found (rule check) | Combine search results when the query is present in multiple successive messages |
| instance_element-hq__element-web-ee13e23b156fbad9369d6a656c827b6444343d4f-vnan | no overlap found (rule check) | `RoomHeaderButtons` can crash when thread notifications are unsupported or when the `room` prop is missing. |
| instance_element-hq__element-web-f14374a51c153f64f313243f2df6ea4971db4e15 | no overlap found (rule check) | Improve Message Composer Component Visibility |
| instance_element-hq__element-web-f3534b42df3dcfe36dc48bddbf14034085af6d30-vnan | no overlap found (rule check) | Membership event combining display name and profile picture changes lacks a single descriptive message |
| instance_element-hq__element-web-f63160f38459fb552d00fcc60d4064977a9095a6-vnan | no overlap found (rule check) | Inconsistent and unclear display of key verification requests in timeline |