Member of Technical Staff, Coding Research

Lead coding agent evaluation

Develop the methods and systems used to measure and improve AI coding agents. The role combines software engineering, research, and evaluation design, with responsibility for reliable measurement across varied software development tasks.

• Own benchmark specifications, scoring methods, rubrics, and evaluation quality standards.
• Lead research projects that measure and improve coding model performance.
• Create datasets, reference examples, and protocols for dependable assessment.
• Analyze model behavior and recurring failure patterns, then translate findings into training or evaluation improvements.
• Build tools for experimentation, data generation, review workflows, and evaluation pipelines.
• Establish reproducible methods and consistent measurement practices.
• Design experiments with researchers, engineers, and applied AI colleagues.
• Contribute technical reports, benchmark studies, and research findings for clients.

Required technical experience

At least three years of experience in software engineering, machine learning, AI research, evaluation, or a related technical field is required. You should have a strong engineering background in Python, C++, or comparable languages.

Experience designing, reviewing, or validating technical assessments, coding tasks, benchmarks, or evaluation methods is important. Familiarity with language models, coding agents, reinforcement learning, model evaluation, or related systems is also required.

You should be able to build tools, automate processes, and improve workflows through systematic experiments. Strong analysis and communication skills are necessary for investigating complex model behavior and explaining findings to varied audiences.

Helpful research background

Experience with advanced AI systems, coding agents, or evaluation research would be valuable. Other useful experience includes independently leading ambiguous technical projects, building large scale datasets or benchmarks, and working with agent workflows, tool use, reinforcement learning, or post training methods.

Publications, open source work, or demonstrated technical leadership in AI or software engineering can support your candidacy. The role suits someone interested in how data, evaluation, and feedback affect model capabilities and comfortable with changing research priorities.

Compensation and benefits

This is a full time remote position with a national annual base salary range of $140,000–180,000. Employees are eligible for equity, and performance based compensation may also be available depending on the role and applicable policies.

Benefits include up to 100% reimbursement of health insurance premiums, paid time off, and a 401(k) plan with an employer match. The selection process uses AI tools to assist screening and assessment while retaining human decision making.