Senior Software Engineering Experts

The work

Help evaluate AI on authentic software engineering work. Draw on senior experience at major technology, quantitative or financial organizations to create demanding tasks and judge generated code using clear evidence.

• Design realistic assignments involving feature changes in large repositories, debugging, root cause analysis, code review, architecture, repository questions, documentation or planning.
• Explain in task proposals why the work is realistic, how it will be verified and what makes it challenging for current models.
• Write reference solutions, tests and criteria covering correctness, architecture, maintainability, style and scope.
• Evaluate generated engineering work and collaborate with coding agents to understand their strengths and failures.
• Make consistent, defensible assessments that other reviewers can reproduce.
• Revise submissions using structured feedback from senior reviewers.

Experience and skills

• The ideal background includes at least four years of software engineering in qualifying organizations; experience may be combined across employers.
• Qualifying settings include major technology or quantitative firms at mid senior level, leading banks, hedge funds or trading firms, or widely recognized growing technology businesses at senior level.
• ML, infrastructure, data, security and quantitative developer roles may also qualify.
• Strong ability in a major language such as Python, Java, C++, Go, TypeScript or Rust, alongside system design, debugging, testing and review skills.
• Experience with large production repositories and code quality assessment beyond whether tests pass.
• Excellent written English for explaining decisions and tradeoffs.
• A bachelor’s degree from an English speaking country.
• Careful, consistent evaluation and openness to feedback and calibration.

Working arrangements

The project accepts candidates based in the United States, United Kingdom, Canada, Australia or New Zealand and supports research on practical AI engineering performance.