Intelligence Analyst

Analytic evaluation for AI

Contribute intelligence analysis experience to a remote contract evaluating AI capabilities. The work examines how generated responses handle evidence, reasoning, uncertainty, and practical usefulness; no previous AI experience is required.

Responsibilities

• Develop analytic cases and map them to defined evaluation taxonomies.
• Write complex test cases that assess source handling, analytic rigor, and appropriate expressions of uncertainty.
• Grade responses for reasoning, source use, recognition of limitations, and practical utility.
• Flag cases where AI unnecessarily refuses legitimate analytic work and explain the issue clearly.
• Document difficult cases and disagreements to improve evaluation methods.
• Help refine source evaluation and confidence assessment procedures with project leads.
• Contribute to consistent analytic standards and reliable AI benchmarks.

Preferred background

• Experience integrating multiple sources, or substantial expertise in a specific analytic discipline, ideally with a civilian intelligence background.
• Strong source evaluation, confidence assessment, and familiarity with established analytic standards.
• Experience teaching, mentoring, or leading analytic reviews.
• Precise written and verbal communication and the ability to explain nuanced model limitations.
• Familiarity with structured analytic techniques and intelligence analysis standards.

Previous AI evaluation or model assessment work is helpful but is not required.