Research Evaluation Specialist (PhD or Masters / Researcher / Professor)

About the work

Develop difficult research questions, test them against AI, and document defensible answers with primary sources. This worldwide remote contract is approximately 15 hours weekly and pays per task meeting specifications; prior AI training or evaluation experience is required.

Responsibilities

• Create original, high-difficulty question-and-answer pairs at the advanced of your discipline to rigorously assess AI model capabilities.
• Source, verify, and thoroughly document answers using primary sources and authoritative references, including clear citations and reasoning.
• Design questions demanding advanced reasoning, methodological nuance, or synthesis across multiple sources, ensuring resistance to oversimplified or shortcut solutions.
• Test questions against AI systems, identify items that are not sufficiently challenging, and iteratively increase complexity while maintaining accuracy.
• Write each question and answer with high precision and clarity, eliminating ambiguity and supporting defensible responses.
• Integrate reviewer feedback and consistently uphold all project guidelines and quality standards throughout your deliverables.

Preferred background

• Masters or PhD completed, active candidacy, or equivalent research experience as a specialist, researcher, assistant professor or professor in your domain.
• Demonstrated track record of scholarly research, in-depth subject matter expertise, and familiarity with primary literature.
• Exceptional analytical thinking, close attention to detail, and strong written communication skills in English.
• Experience sourcing and triangulating information from authoritative references.
• Proven self-direction and reliability in delivering expert-level work independently within remote engagements.
• Skill in composing challenging, original, and methodologically sound questions.
• Prior experience in AI training, data labeling, or model evaluation is required.

Payment and availability

Payment is based on tasks meeting project specifications; task duration varies by experience and workflow. Weekly minimum submission requirements apply, although the listing does not specify a count.

Immediate availability is preferred. Selected contributors are expected to start their first tasks within 24 to 48 hours of completing onboarding.