SWE-bench_Verified princeton-nlp
Dataset Summary SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality. SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. See this post for more details on the human-validation process. The dataset collects 500 test Issue-Pull Request pairs from popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution. The original… See the full description on the dataset page: https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified.
- 类型
- dataset
- 下载量
- 397,985
- 点赞
- 380
- 访问
- public
- 文件
- 0
标签
- 评测基准
- 代码
- 代码生成
- Agent
- 大语言模型
摘要
SWE-bench Verified 是从 SWE-bench 测试集中抽取的500条人工验证质量的子样本，用于评测AI系统自动解决GitHub issue（问题）的能力。数据集包含500对"问题-拉取请求"样本，来自热门Python仓库，通过单元测试验证（以PR后行为为参考解）进行评测。适用于大语言模型与Agent在真实软件工程场景下的代码修复、问题解决能力基准测试。
README
--- dataset_info: features: - name: repo dtype: string - name: instance_id dtype: string - name: base_commit dtype: string - name: patch dtype: string - name: test_patch dtype: string - name: problem_statement dtype: string - name: hints_text dtype: string - name: created_at dtype: string - name: v…