Leaklens

SWE-bench Lite: every task, with its verdict

Download CSV
TaskVerdictIssue title
pytest-dev__pytest-5227Tier A: the fix is stated in the issueImprove default logging format
pytest-dev__pytest-5413Tier A: the fix is stated in the issuestr() on the pytest.raises context variable doesn't behave same as normal exception catch
pytest-dev__pytest-7168Tier A: the fix is stated in the issueINTERNALERROR when exception in __repr__
pytest-dev__pytest-8906Tier A: the fix is stated in the issueImprove handling of skip for module level
pytest-dev__pytest-11143Tier B: the issue points to the file or function that was changedRewrite fails when first expression of file is a number and mistaken as docstring
pytest-dev__pytest-7373Tier B: the issue points to the file or function that was changedIncorrect caching of skipif/xfail string condition evaluation
pytest-dev__pytest-7432Tier B: the issue points to the file or function that was changedskipping: --runxfail breaks pytest.mark.skip location reporting
pytest-dev__pytest-11148no quoted evidenceModule imported twice under import-mode=importlib
pytest-dev__pytest-5103no quoted evidenceUnroll the iterable for all/any calls to get better reports
pytest-dev__pytest-5221no quoted evidenceDisplay fixture scope with `pytest --fixtures`
pytest-dev__pytest-5495disputedConfusing assertion rewriting message with byte strings
pytest-dev__pytest-5692no quoted evidenceHostname and timestamp properties in generated JUnit XML reports
pytest-dev__pytest-6116disputedpytest --collect-only needs a one char shortcut command
pytest-dev__pytest-7220no quoted evidenceWrong path to test file when directory changed in fixture
pytest-dev__pytest-7490disputedPytest 6: Dynamically adding xfail marker in test no longer ignores failure
pytest-dev__pytest-8365no quoted evidencetmpdir creation fails when the username contains illegal characters for directory names
pytest-dev__pytest-9359no quoted evidenceError message prints extra code line when using assert in python3.9