SWE-Bench Task Auditor

Benchmark task auditing

Assess software engineering tasks that train and evaluate an AI lab's models. Review repository level assignments, reference patches, test harnesses and the integrity of grading, then provide clear written findings grounded in the rubric.

Required experience

• At least 3 years of professional software engineering.
• Demonstrated open source contributions or maintenance, such as merged pull requests or committer responsibilities.
• Ability to inspect reference patches, test runners and Docker isolation and identify answer leakage or reward hacking.
• Fluency in Python and at least one of Java, Go, TypeScript or C++.

Preferred background

Experience with SWE-Bench Verified or related repository benchmarks is useful. Maintainer work on major Python projects such as Django, Flask, scikit-learn, sympy or pytest is valued, as is prior code review or task grading.