Senior Software Engineer — AI Evaluation & Benchmarks

Evaluation infrastructure

Build coding benchmarks and data pipelines used to measure AI performance on practical programming tasks. The work examines whether models can reason, debug, and produce software that meets production standards across large repositories and multiple languages.

• Design and implement tests for real-world coding tasks.
• Build and maintain scalable data processing for model evaluation.
• Analyze generated code for correctness, reliability, and failures on unusual inputs.
• Develop structured scenarios that test debugging, reasoning, and software quality.
• Work in large code repositories and environments involving several languages.
• Contribute to systems and frameworks that improve AI understanding of software.
• Give detailed feedback on model performance and recurring failure patterns.

Required background

At least four years of professional software engineering experience is a firm requirement. You must have worked in a rapidly growing technology business or a highly regarded software organization, with experience delivering large products or systems in demanding engineering environments.

Expert Python proficiency is essential, including clean, efficient, well-tested implementations. Practical experience with large and complex repositories, Git, and modern development processes is required.

You must have already designed and implemented LLM coding benchmarks and data pipelines. Native or bilingual English with strong writing, technical rigor, and independent ownership are also required.

Additional qualifications

Senior or lead engineering experience with substantial technical ownership is particularly relevant. A bachelor's or master's in Computer Science, Machine Learning, or a related area, or equivalent professional experience, is welcome.

Additional languages such as JavaScript, Go, or C++ are useful. Experience with CI/CD and robust tests using pytest, Mocha, or JUnit is also valuable, alongside security engineering or significant open-source contributions.

Familiarity with AI or ML evaluation methods and benchmarking further supports the work.

Contract terms

The engagement is an hourly, fully remote contract lasting three months, with full-time availability preferred. A reliable internet connection is needed for remote work.

You collaborate internationally with engineers and researchers on evaluation systems that influence how models are measured and improved. Further challenges may lead to a contract extension or continuing engagement.