SWE-Bench Pro (public sample): every task, with its verdict
SWE-Bench Pro, published by Scale AI. Paper authors: Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu and Brad Kenstler. SWE-Bench Pro (arXiv) · This page covers a random 100-task sample of the 731 public tasks (v1 config of the ScaleAI/SWE-bench_Pro dataset). The sample is the 100 instance IDs with the lowest SHA-256 of "own1:swe-bench-pro:<instance_id>", so anyone can redraw it. · Report a correction · flags show what the public text and git history contain: public exposure was possible, nothing more.
| Task | Verdict | Issue title |
|---|---|---|
| instance_flipt-io__flipt-2ca5dfb3513e4e786d2b037075617cccc286d5c3 | no overlap found (rule check) | Support multiple metrics exporters (Prometheus, OpenTelemetry) |
| instance_flipt-io__flipt-2eac0df47b5ecc8bb05002d80383ceb08ab3620a | no overlap found (rule check) | Telemetry payload doesn't reflect analytics state or backend and carries an outdated payload version identifier |
| instance_flipt-io__flipt-756f00f79ba8abf9fe53f3c6c818123b42eb7355 | no overlap found (rule check) | Configuration refactoring to separate warnings from Config and deprecate ui.enabled |
| instance_flipt-io__flipt-86906cbfc3a5d3629a583f98e6301142f5f14bdb-v6bea0cc3a6fc532d7da914314f2944fc1cd04dee | no overlap found (rule check) | Snapshot cache does not allow controlled deletion of references |
| instance_flipt-io__flipt-aebaecd026f752b187f11328b0d464761b15d2ab | no overlap found (rule check) | Git backend fails to poll when a reference no longer exists |
| instance_flipt-io__flipt-b2cd6a6dd73ca91b519015fd5924fde8d17f3f06 | no overlap found (rule check) | Telemetry warns about non-writable state directory in read-only environments |
| instance_flipt-io__flipt-c12967bc73fdf02054cf3ef8498c05e25f0a18c0 | no overlap found (rule check) | Context cancellation and deadline exceeded errors are incorrectly classified as internal in gRPC responses. |
| instance_flipt-io__flipt-c6a7b1fd933e763b1675281b30077e161fa115a1 | no overlap found (rule check) | Add namespace and version metadata to export files; validate on import |
| instance_flipt-io__flipt-f36bd61fb1cee4669de1f00e59da462bfeae8765 | no overlap found (rule check) | `flipt validate` produces imprecise and repetitive error messages when validating YAML files. |