Remote Python Senior Software Engineer, AI Training

Evaluate AI coding interactions

Assess how coding agents behave in realistic development situations, with attention to the engineering judgment behind their responses. The work focuses on reviewing interactions rather than delivering production code. Evaluate whether the guidance is sensible, useful and credible to an experienced developer.

What you will assess

• Review complete AI coding exchanges and judge whether the responses are useful, broadly correct and consistent with strong engineering practice.
• Evaluate the opening explanation, reasoning and guidance as well as the code itself.
• Distinguish between different quality levels and explain why one response deserves a stronger rating than another.
• Write clear, direct feedback identifying what worked, what failed and what seemed misleading or inappropriate.
• Help establish expectations for effective interactions with tools such as OpenAI Codex, Claude Code and Cursor.

The judgment involved goes beyond valid syntax. Consider whether explanations help the user, whether the agent guides the work effectively and whether an interaction increases or undermines trust. You should be comfortable defending subjective assessments with rigorous reasoning.

Experience required

• Senior, Staff or Principal engineering experience, or an equivalent level of responsibility.
• A strong background in TypeScript/JavaScript or Python.
• Direct experience with at least one of OpenAI Codex, Claude Code or Cursor, plus deep familiarity with development workflows that involve AI assistance.
• The ability to judge code without running it or examining every individual line.
• Written and spoken English at B2 level or higher; the work includes detailed written assessments and brief video explanations.
• Confidence in giving direct opinions and a demanding standard for engineering quality.

Experience designing prompts or working on evaluations would be useful. Mentoring senior engineers or helping define engineering standards is also advantageous, but neither is mandatory.

Contract and selection

This is remote contract work with worldwide reach, although some locations may not be supported by the client. Compensation is $100 to $200 per hour, determined by experience and location, for a flexible commitment of 10 to 20 hours per week.

The engagement is ongoing, with individual projects lasting roughly two weeks to a few months. Evaluators who perform well may be offered further projects.

Selection involves one take home evaluation exercise and a recorded Loom walkthrough, with no interview. Work can begin once the exercise is passed and a project place is available.