PhD Rater

1\. Role Overview Looking for experienced researchers and technical experts to contribute to a project supporting a frontier-model evaluation effort focused on agentic workflows. You’ll design and validate challenging benchmark tasks in data science, machine learning, finance, and coding to help surface and diagnose reasoning and problem-solving gaps in a target STEM model. The work centers on building robust, real-world tasks with executable tests and then analyzing model/agent behavior.

2\. Key Responsibilities

• Design challenging, real-world STEM problems
• Implement each task inside an agentic development environment using Python

3\. Core Qualifications

• Deep expertise in data science, machine learning, finance, and/or Python-based coding
• Active or recently graduated PhD (Top U.S.-based school)
• Strong research background in frontier STEM topics
• Ability to engage reliably for 30+ hours/week, primarily on weekdays
• Demonstrated technical output such as high-quality open-source contributions (especially in agentic / LLM tooling ecosystems)
• Comfort reading and reasoning about agent behavior traces to diagnose failure modes beyond surface-level errors

4\. More About the Opportunity

• Initial focus area: agentic workflows for STEM tasks
• Familiarity with agentic frameworks and OSS ecosystems is helpful (examples include LangChain, MetaGPT, AutoGen, AutoGPT, CrewAI, LlamaIndex, BabyAGI, SuperAGI, CAMEL, AgentGPT, Dify, etc.)
• Deliverables are expected to be reproducible and testable (clear specs, deterministic tests where possible, documented environments)

5\. About - is a talent marketplace that connects top experts with leading AI labs and research organizations.

• Our investors include Benchmark, General Catalyst, Adam D’Angelo, Larry Summers, and Jack Dorsey.
• Thousands of professionals across domains like law, creatives, engineering, and research have joined to work on frontier projects shaping the next era of AI.