Senior Software Engineer – AI Evaluation

AI evaluation infrastructure

Build and scale software that helps research teams understand AI model capabilities, failure modes, and opportunities for improvement. This remote hourly contract combines software engineering with AI research and calls for 20 to 40 hours of work per week.

The work includes tools, pipelines, and frameworks that make evaluation repeatable and help teams compare results. Scheduling is flexible, with independent work and asynchronous collaboration.

Responsibilities

• Design scalable evaluation pipelines and frameworks across different tasks and domains.
• Build automated testing harnesses, scoring systems, and benchmarks for large language models and other AI systems.
• Write production-quality code to process, analyze, and visualize evaluation datasets at scale.
• Maintain APIs, dashboards, and internal tools for running, tracking, and comparing evaluations.
• Translate researchers' and data scientists' evaluation methods into reliable, repeatable software.
• Identify edge cases, failure modes, and reliability issues in model outputs.
• Improve system performance, data processing speed, and infrastructure costs.
• Contribute to architecture and technical direction of the evaluation system.
• Document technical work and participate in code reviews to maintain engineering quality.

Required experience

• At least five years of professional software engineering, with evidence of building and shipping production systems.
• Strong Python proficiency, including pandas and NumPy, plus experience with FastAPI, Flask, Django, or similar web frameworks.
• Solid knowledge of architecture, design patterns, and software engineering practices.
• Experience handling large datasets and building data pipelines.
• Comfort with AWS, GCP, or Azure and containerized deployments.
• Familiarity with Git, CI/CD workflows, and testing frameworks.
• Strong problem-solving skills and the ability to work through ambiguous requirements independently.
• Clear written communication and reliable work habits in a remote, asynchronous environment.

Helpful background

Experience with ML or AI evaluation, benchmarking, and model testing is valued. Other useful experience includes large language models, prompt engineering, AI safety and alignment concepts, developer tools, internal systems, and data infrastructure.

Distributed systems, message queues, workflow orchestration such as Airflow or Prefect, and statistical methods for comparing model performance are also relevant. Prior remote or asynchronous engineering experience and open-source contributions to AI, ML, or evaluation tools are helpful.

Contract outlook

The work offers collaboration with engineers and researchers and exposure to AI capabilities and evaluation methods. Ongoing work and contract extension may become available as projects expand.